CrawlTwitNumbers - My Latest Project

https://github.com/ScriptProdigy/CrawlTwitNumbers

Being a twitter user myself, I see a LOT of my friends post their numbers on twitter. So, I had a thought a couple weeks ago to data mine those numbers. I'm using twitters public stream api for data mining the numbers. Twitter provides an extensive API for programmers for all means. They allow data mining and such, its pretty awesome really. Anyways, Check out the git hub for the source and read the readme to start it up yourself. Also, it isn't finished yet but expect one that crawls for basic patterns in the next few hours after this post. Once I get the basics down I'll probably add a bunch of features like logging the location of the person and such.

Basic phone number searching implemented, a fuck ton of false positives because people like putting long as numbers in their name and tweeting random numbers.. Ugh

You're just grabbing random number sequences from tweets You need to improve on that regular expression pattern, haha.

Also, .gitignore:

*.py[cod]



# C extensions

*.so



# Packages

*.egg

*.egg-info

dist

build

eggs

parts

bin

var

sdist

develop-eggs

.installed.cfg

lib

lib64

__pycache__



# Installer logs

pip-log.txt



# Unit test / coverage reports

.coverage

.tox

nosetests.xml



# Translations

*.mo



# Mr Developer

.mr.developer.cfg

.project

.pydevproject[/cod]

Make it check to see if the number is longer than 10 chars and stuff like that to cut down on false positives. Like, make it grab the entire chunj that has no spaces

(EG if I post 11122233334, your thing will take the entire sequence and reject it because of the extra four. If it sees 1112223333hmuuuu, it will still take in the entire line but only save the numbers because its still the correct amount of numbers)

Artificial wrote:

You're just grabbing random number sequences from tweets You need to improve on that regular expression pattern, haha.

Also, .gitignore:

*.py[cod]



# C extensions

*.so



# Packages

*.egg

*.egg-info

dist

build

eggs

parts

bin

var

sdist

develop-eggs

.installed.cfg

lib

lib64

__pycache__



# Installer logs

pip-log.txt



# Unit test / coverage reports

.coverage

.tox

nosetests.xml



# Translations

*.mo



# Mr Developer

.mr.developer.cfg

.project

.pydevproject[/cod]

Found it on google

I've been with an eye on this data miner, it's really cool =] ' really nice. I think you should do more stuff for it, but people doesn't post more ideas, but it's a smart open source ^^

Dudutxsuio wrote:

I've been with an eye on this data miner, it's really cool =] ' really nice. I think you should do more stuff for it, but people doesn't post more ideas, but it's a smart open source ^^

I'm always getting swamped with different projects lol, more recently then ever. Appreciate someone that digs my stuff though! Its kind of difficult sorting numbers when random accounts tweet out random strings of digits and the software catches that all the time