Data Analysis -- Time Series Data

A recent project I've been working on is scraping 4chan's boards and iterating through each post. For each post I'll tolkenize the text with nltk and find proper nouns, then get a frequency (sum of all posts which mention the proper noun) for a certain epoch. I've got the data figured out (besides some of the tolkenizing and classification), but I'm trying to visualize it in something like d3.js and am having some issues. The stack is a python script which does the scraping/tolkenizing and puts the data into a csv file named something like board-epoch.csv. This is run by a cronjob every 30 minutes or so. I have a golang webapp that's combining these generated csv files into one big csv that can be referenced by frameworks like d3.js. So the data format is essentially one big csv that looks like this (/g/ as an example board) with the fields as the word, frequency, and epoch respectively: Linux,42,1515991999.777177 Android,30,1515991999.777177 Python,21,1515991999.777177 Google,14,1515991999.777177 Sorry,13,1515991999.777177 Apple,12,1515991999.777177 [...] Linux,44,1515992401.907241 Android,30,1515992401.907241 Python,21,1515992401.907241 Google,14,1515992401.907241 Sorry,13,1515992401.907241 Apple,12,1515992401.907241 [...] Linux,46,1515993602.477694 Android,29,1515993602.477694 Google,15,1515993602.477694 Firefox,14,1515993602.477694 Sorry,13,1515993602.477694 Apple,12,1515993602.477694 Python,11,1515993602.477694 I just have some questions that someone might be able to answer: 1. What is the best practices for storing data like this? I'm not completely tied to this format of keeping data since I'm sure there's better best practices out there. I'd be using the words as keys. Any time the key would have no data would be treated as a frequency of 0 (like the off-chance an hour goes by where linux isn't mentioned on a single thread!) 2. What's this analysis or type of data called? Even getting a name for what this sort of data or analysis is called should help me figure this out. I'm not a statistician or mathematician, or even versed in science. 3. How would you recommend visualizing this? I've used prometheus+graphana at work but would like to use something a bit more pretty and embed-able in a webpage. I've played with d3.js for a couple days but haven't groked it yet, especially since I'm not experienced with javascript. I'm looking for something more "plug-and-play" as in I'd give the data to it in a recognized format and have it do the work for me, but I'm up for a challenge too. 4. Anyone know of any tolkenizers specifically for social media/imageboards? I'm using the nltk.pos_tag and it's not all that useful since I'm pretty sure it's not tracking lowercase nouns ("Linux" compared to "linux") and wasn't trained against that kind of corpus. Again I've never formally learned NLP or ML but any points in the direction are appreciated. 5. Has anything like this been done before? attached pic is a prototype that will take one "frame" (epoch) of this data and visualize it with some simple python library. The reason for this project if anyone's curious is to visualize trends in 4chan, to spot things like trends in media tastes (lit, mu, tv, etc.) shills (ck, g, etc.) and just have some fun and experience with this sort of stuff. I've been on that godforsaken place for most of my childhood and life now so it'd be nice to understand it better. If I can come up with a basic framework and data format there's a lot that could be done with it. Thanks for reading and hopefully you can help out!

Just figured I'd update that I think I'll be creating maps which have a key of the word (in the picture's example "Star Wars") and a value of a 2d array like { [freq1, epoch1], [freq2, epoch2] ... }. d3 should be able to handle these maps and if not it shouldn't be too hard to demarshall them.

Just figured I'd update since I've gotten a lot farther. I'll be going with: Python -> SQL-ish -> Grafana. With whatever DB I'd use, there'd be separate tables for each board (4chan_g, 4chan_tv, etc.), rows of timestamps, and columns of the different words. I have experience with grafana and realized I could embed it in webpages via iframe which fulfills my requirement of being viewable in a standard webpage. The columns would have high cardinality and postgresql maxes out at < 1600, but there's definitely other options out there that could handle it. What I'm looking for is a "Wide Column Database", probably a docker container of cyanite/graphite since it works well with grafana. I know this isn't my personal blog so mods please close this thread if this sort of updating/bumping is discouraged.

First thing I can think of on visualizing this is word clouds. It would be cool to have some sort of animated word clouds showing board trends.

godspeed anon https://bokeh.pydata.org/en/latest/

>>6304 This is wonderful and I think will solve a lot of my problems, especially since I can keep most of the stack python. Maybe some flask or django can get this live and publicly-viewable. I'll still likely need some nice wide-column database software that can interface well with whatever webserver I'll be using but I'm sure that it will be easier than finding something that plugs into grafana. Thanks for the help! I'll give it a shot and inform yall if this goes well.

PROGRESS! I have the pipeline finally set up to go from the scraper to a database. IDK why I couldn't figure out the database schema, I was thinking it was more complicated than it was. Now it's just a matter of getting things from postgresql to bokeh.

I got the data in panda frames and then into a bokeh graph! In this format it'll be a lot easier to do analysis on. Especially when I get about a month worth of data. Until then!

>>6172 pls post source senpai