jlengrand/twitterboard
1
0
Fork 0
mirror of https://github.com/jlengrand/twitterboard.git synced 2026-03-10 08:51:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
139 Commits 7 Branches 0 Tags
3277ebdffec38bb331c4d85b9e134a57f3ec9bc0
Commit Graph

139 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Julien Lengrand-Lambert
3277ebdffe In the process of adding the list of current active hashtags on web page 
Creating a table in javascript is not a good way, as it is not rendered correctly in the browser.
Better format it server side or find a way to populate it correctly client side.

Also has to create a remove_hashtag method that takes arguments in input
 2013-02-17 19:39:37 +01:00
Julien Lengrand-Lambert
2e8a425cb7 Structures the html templates a bit more and adds categories in the page. 
NExt steps:
- List all currently tracked hashtags on main page
- Add icon to remove those hashtags next to them.
 2013-02-17 18:15:48 +01:00
Julien Lengrand-Lambert
ab12c04371 Removes test file flask_stop from repo. 
Adds method remove_hashtag to flask_app.

Next step is to generate a list of currently tracked hashtags in the main webpage
 2013-02-17 17:45:52 +01:00
Julien Lengrand-Lambert
3e37007cb1 Adds flask to the list of requirements. 
It was missing for some reason.

add_hashtag is now wokring in flask_app.
It will later have to be changed though, as the current version is evrything but RESTful.

Next step is to display a list of currently tracked hashtag in the application.
 2013-02-17 17:41:31 +01:00
Julien Lengrand-Lambert
913e1fde1b First version working in flask_stop. Add_hashtag and remove hashtags methods seem to work ok 2013-02-17 15:31:21 +01:00
Julien Lengrand-Lambert
470a56e911 Might I be on the right track? Finally ? 2013-02-15 00:32:18 +01:00
Julien Lengrand-Lambert
949d5979c0 Another typo 2013-02-05 11:33:02 +01:00
Julien Lengrand-Lambert
f27aa1794d Corrects typo 2013-02-05 11:21:32 +01:00
Julien Lengrand-Lambert
72d0203b83 Adds added hashtag message 2013-02-05 11:17:29 +01:00
Julien Lengrand-Lambert
1594fb2901 updates flask_app 2013-02-05 11:09:01 +01:00
Julien Lengrand-Lambert
058766a0f5 statistics now works fine 2013-02-05 11:07:45 +01:00
Julien Lengrand-Lambert
004edca34a statistics now works fine 2013-02-05 11:05:03 +01:00
Julien Lengrand-Lambert
4fec192939 First working version of hashtag adding in html. To be merged 2013-02-05 10:59:49 +01:00
Julien Lengrand-Lambert
8f7d576a8f starts creating the form to add hashtags to track 
Not functional yet
 2013-02-05 09:57:45 +01:00
Julien Lengrand-Lambert
2311503642 Creates offline mode for the streamer to avoid connecting to Twitter APi if not needed 2013-02-05 09:20:51 +01:00
Julien Lengrand-Lambert
99bf3c3d2a Adds new streamer_status option in data file. 
Used to avoid connecting to twitter streaming API during web development.
 2013-02-05 09:11:19 +01:00
Julien Lengrand-Lambert
2eb3d215bc Removes timeout connexion to debug flask app 2013-02-05 08:35:11 +01:00
Julien Lengrand-Lambert
e6c7751fb8 Attempt to reduce number of database connections to the database coming from Flask 2013-02-04 19:28:55 +01:00
Julien Lengrand-Lambert
4e38990bbb Comments unused library 2013-02-04 16:57:09 +01:00
Julien Lengrand-Lambert
f35f2006c5 Upgrades statistics pages to give number of tweets in database and members. 
Corrects datamodel to name all tables in a singular form.
All tables have to be removed and recreated from scratch to work properly now.
 2013-02-04 16:46:55 +01:00
Julien Lengrand-Lambert
374859650c Now only counts active hashtags in webpage. 
Also removes connection to tweeter streaming API during client development
 2013-02-04 16:34:45 +01:00
Julien Lengrand-Lambert
180f30af07 Adds proper session closing and engine disposal in flask_app. 
Adds sme missing statements in counter for deprecated methods.
 2013-02-04 16:26:14 +01:00
Julien Lengrand-Lambert
e2dc517450 Updates flask_app to provide it with generic database url 2013-02-04 14:02:11 +01:00
Julien Lengrand-Lambert
440e938f57 Merges core part with web app part to avoid having multiple dependencies. 
Also seems more logical.
 2013-02-04 13:52:05 +01:00
Julien Lengrand-Lambert
d3e70ea3a0 Adds connection to database lost error found in PythonAnywhere. 
Probably comes from them, but it should not crash everything.
 2013-01-31 19:16:03 +01:00
Julien Lengrand-Lambert
f374382032 Adds imit to leader query. 
this with addition of an index should dramatically reduce query processing time and increase performance.

Next step is probably to join all queries together and parse in Python afterwards.

TODO:
Start working on rendering again
Try to scale with a hundred hastags.
 2013-01-31 16:27:09 +01:00
Julien Lengrand-Lambert
5af74eb10c Adds another index on member table, aiming at hashtags and counts in order to generate the leaderboard faster. 2013-01-31 14:23:47 +01:00
Julien Lengrand-Lambert
b16c64e2c4 Adds file check for ini file. 
Now exists gracefully if file is not found.

Avoids ugly errors on first launch when deploying to a new machine
 2013-01-31 14:16:30 +01:00
Julien Lengrand-Lambert
b504b2acf4 Reenables tweet logging. 
As tweets are now just logged and not crawled any more, insertion time in the database should not be an issue any more.
 2013-01-31 14:05:49 +01:00
Julien Lengrand-Lambert
e3f12fb597 Modifies code so that Indexes are really taken into account. 
Tested full night without increasing CPU load.
 2013-01-30 07:51:16 +01:00
Julien Lengrand-Lambert
9432d2ca86 Creates indexes to enhance searches in table . 
Also activates logging on mysql to look at queries and try to correct if
possible.
 2013-01-29 16:52:42 +01:00
Julien Lengrand-Lambert
025c4a7502 Removes small bug finding duplicates where there is none. 
Also removes print statement from data.py
 2013-01-28 16:45:55 +01:00
Julien Lengrand-Lambert
e27c53db82 Stops logging tweets and directly jumps to hashtags. 
Will se later for an efficient way to save tweets too.
Probably a way that doesnt require database.
 2013-01-27 18:18:32 +01:00
Julien Lengrand-Lambert
ca833e8616 Stream and StreamSaverListener are also creating rogue connections to the database when adding/removing hashtags. 
This had to be taken care of.

Currently is only quick fix, but a better fix should be provided soon.
 2013-01-23 21:04:52 +01:00
Julien Lengrand-Lambert
50239be6c6 Uses same trick for leader as for counter. 
Reduces the number of simultaneous database connections needed to run the full service.
 2013-01-23 20:50:45 +01:00
Julien Lengrand-Lambert
a4749ca591 Attempt to solve max_user_connections issue on mysql in counter 
Also attempts to solvedata.oauth being a string instead of a boolean
 2013-01-23 16:17:16 +01:00
Julien Lengrand-Lambert
64b7ad819c Adds session.close() that had been forgotten in counter.py 
The number of mysql conenctions should decreqse now.

TODO:
Future thoughts to dramatically increase this : Use a database pool.
Threads would simply add new write/commit requests and only one entity actually accesses the database.

This would probably reduce the amount of problems of concurrent connections.
 2013-01-23 10:00:19 +01:00
Julien Lengrand-Lambert
176e4b59ab Removes leftover print in data.py 2013-01-23 09:34:11 +01:00
Julien Lengrand-Lambert
04badb7eff Adds log for encoding errors in MySQL. 
To be fixed soon
 2013-01-23 09:27:12 +01:00
Julien Lengrand-Lambert
7cc5a74ecb Separates encoding scheme for database connexion in ini file. 2013-01-23 09:25:50 +01:00
Julien Lengrand-Lambert
772806d01a Corrects some unicode problems in datamodel 
Adds unicode support in mysql database connexion.
Removes all try/catch in commits that were used to avoid SQLite limitations.

FIXME: There is still some problems with special characters handling in MySQL, but it doesnt cause any problem for the leaderboard printing so far.
It has to be fixed though.

TODO:
Add separate encoding parameters in basic.ini
 2013-01-23 09:19:40 +01:00
Julien Lengrand-Lambert
86b9c48f79 Adds length to String cloumns in datamodel to fit with SQLite requirements. 
A limit of 200 characters has been put for each string.
It should be way more than needed (maximum tweet size is 140 characters)
 2013-01-23 08:51:25 +01:00
Julien Lengrand-Lambert
a945b2399c Installs python eff to work with MySQL 2013-01-22 22:53:08 +01:00
julien Lengrand-Lambert
9c119ee1fd Create command_error_3 
Another error. 

New, this time !
 2013-01-22 06:47:53 -08:00
julien Lengrand-Lambert
a6e8d8301b Create leader_problem_3 
Error of database locked again
 2013-01-22 05:20:43 -08:00
Julien Lengrand-Lambert
ff05f492bb removes .ini files from repo. Changes leaderboard printing back to html 2013-01-22 11:30:24 +00:00
Julien Lengrand-Lambert
35e2999c18 Implements automatic way to find config file without providing path 
Removes database files from repo.
 2013-01-22 12:21:47 +01:00
Julien Lengrand-Lambert
ddd9171d0b Now get 420 errors. 
INvestigating that.
 2013-01-22 11:58:14 +01:00
Julien Lengrand-Lambert
e470ad5617 Finishes converting leader to new config format. 
Removes older config format.
Test now before adding commandline argument
 2013-01-22 11:36:17 +01:00
Julien Lengrand-Lambert
bb4093be02 converts older use of data in leader to new format. 
TODO: Add new variables
 2013-01-22 11:29:21 +01:00