Tag: MySQL

Week 6: Pre-Mortem

--Originally published at TC3045 – Sagnelli's blog

This week is all about joining forces. So many thing have beeen worked on separately, and its time to integrate everything into a first impression of our project. We can now focus on gathering and mining information to be ready for the next phase. The next phase is actually analyzing data, and displaying results of our application. However, that’s a topic for another day.

In the meantime, let’s wait to see how things are developing within our project. Stay tuned for this week’s post-mortem to know what’s going on.

Cheers!

Week 5: Post-Mortem

--Originally published at TC3045 – Sagnelli's blog

As you know we already had a streamer to use with the Twitter API for streaming tweets. With that streamer we were only able to filter using key words; however, tweets from all sources arrived. Now, added a second streamer, where we are now able to filter from whom we want our tweets. This list of users is provided via a .txt file, and we can also use a list of keywords for a more thorough filtering.

What did we do with the first streamer?

It is now fully working, adding filtered tweets to the database, and printing who tweeted the matching results.

The project’s development is going great.

Stay tuned for next weeks developments.

 

Week 4: Post-Mortem

--Originally published at TC3045 – Sagnelli's blog

Hey! Good to see you again.

This week I added environment variables for the connection information on the database. Also I changed some classes names, and improved the code for better understanding. Connection to the database is working, as well as insertion. However, we still have problems with encoding the text to UTF-8, and avoiding weird characters. EMOJIS are a big problem because, whenever a user has an EMOJI in his user name, or in his tweet, the program halts.

We are still figuring a way to store the text with the emojis for further sentiment analysis. I know it should be possible.

Stay tuned.

Week 4: Database & Python 4 life

--Originally published at TC3045 – Sagnelli's blog

What’s new, it’s Mike again.

This week is going to be about improving what I previously achieved on the project. I will fix the code, applying some better practices while coding, and I will add some functionalities. Reading, and deleting from the database shall be implemented. Also, I will try to store the tweets without modifying the text for further use on sentiment analysis. For this last part, everything matters; from emojis, to raw text. Henceforth, I will be researching how to store tweets with special characters, and emojis into the database.

This is what I will try to do, and I will keep learning about Python, MySQL, and all these new topics for me.

Stay tuned for what I achieve this week.

From now on, bad practices shall be avoided.

Post-Mortem: Tweets buffer

--Originally published at TC3045 – Sagnelli's blog

Sup, it’s me again.

What I did this week was not so different from what I had planned. However, I did not fulfilled what I was thinking about. I did something a little bit different, but it was a nice progress towards the final project.

I was able to add tweets’ raw text to a list in Python, and when a specified number of tweets are in the list, those tweets are stored in a database.

I was able to do so by modifying the streamer.py, and adding the tweets to a list there. Afterwards, by calling the function in the micro-service storage.insert-tweet.py to do the insertion of whatever the number of tweets is specified.

I had trouble when the text had break lines, unknown characters, or Emojis. Henceforth, I had to modify the text a little to be able to store them in the database.

Up next is adding  some environment variables for the database connection, and adding some update, and delete functions to the application.

Week 3: PyMySQL is about to be applied

--Originally published at TC3045 – Sagnelli's blog

“When you want to succeed as bad as you want to breathe, then you’ll be successful.”

– Eric Thomas

Hey, I’m still awake. Now, there’s a reason to it, and as bad as I want that reason to be watching a Netflix series, or playing videogames, it is not. I am thinking on what I will be doing this week for the Elections Analyzer 2018 project.

This week is going to be all about creating generic functions in Python for DML queries on a MySQL database. I will be dividing this functions as micro-services. Henceforth, there is going to be a separate micro-service for inserting (creating), selecting (reading), updating, and deleting, (CRUD), from a generic database. Everything using the PyMySQL library.

Stay tuned for the post-mortem update of this week’s development.

Update – Elections Analyzer 2018 improvements

--Originally published at TC3045 – Sagnelli's blog

This week a member of the team automatized the setup, and run methods of the application. As this is a two part project, we are focusing on the python micro-services of gathering, cleaning, and storing data in our database.

Database

We are using a JSON to normalize data for our relational MySQL database. We already discovered how to establish connection from Python to a MySQL database using the PyMySQL library to make DDL & DML queries.

This is an example of the code to do so.

from __future__ import print_function
import pymysql

conn = pymysql.connect(host='', port=, user='', passwd='', db='',autocommit=True)
cur = conn.cursor()

#cur.execute("CREATE TABLE Partidos ( ID int NOT NULL, nombre varchar(50), PRIMARY KEY(ID)); ")
cur.execute("INSERT INTO Partidos VALUES (111,'PAN','IZQ')")
cur.execute("INSERT INTO Partidos VALUES (112,'MORENA','DER')")
cur.execute("INSERT INTO Partidos VALUES (113,'PRI','IZQ')")
cur.execute("INSERT INTO Partidos VALUES (114,'MOVIMIENTO CIUDADANO','IZQ')")
cur.execute("SELECT * FROM Partidos")
cur.execute("DELETE FROM Partidos")
print()
for row in cur:
    print(row)
cur.close()
conn.close()

 

This is what I’ve done so far in the project, and I learned how to use micro-services in Python. I will continue doing generic automatization of queries for when the database is up and running.

Keep tuned for further news on the development of the project.

Update – Elections Analyzer 2018 improvements

--Originally published at TC3045 – Sagnelli's blog

This week a member of the team automatized the setup, and run methods of the application. As this is a two part project, we are focusing on the python micro-services of gathering, cleaning, and storing data in our database.

Database

We are using a JSON to normalize data for our relational MySQL database. We already discovered how to establish connection from Python to a MySQL database using the PyMySQL library to make DDL & DML queries.

This is an example of the code to do so.

from __future__ import print_function
import pymysql

conn = pymysql.connect(host='', port=, user='', passwd='', db='',autocommit=True)
cur = conn.cursor()

#cur.execute("CREATE TABLE Partidos ( ID int NOT NULL, nombre varchar(50), PRIMARY KEY(ID)); ")
cur.execute("INSERT INTO Partidos VALUES (111,'PAN','IZQ')")
cur.execute("INSERT INTO Partidos VALUES (112,'MORENA','DER')")
cur.execute("INSERT INTO Partidos VALUES (113,'PRI','IZQ')")
cur.execute("INSERT INTO Partidos VALUES (114,'MOVIMIENTO CIUDADANO','IZQ')")
cur.execute("SELECT * FROM Partidos")
cur.execute("DELETE FROM Partidos")
print()
for row in cur:
    print(row)
cur.close()
conn.close()

 

This is what I’ve done so far in the project, and I learned how to use micro-services in Python. I will continue doing generic automatization of queries for when the database is up and running.

Keep tuned for further news on the development of the project.