I decided to create something to track cryptocurrency Twitter accounts and to detect trends (and offer alerts on this data on Cryptalert).
So we will create a Laravel Zero application and integrate /laravel-console-dusk, a custom version of Laravel Dusk for Laravel Zero, to easily track the evolution of twitter accounts.
What is Laravel Zero ?
Laravel Zero was created by, and is maintained by Nuno Maduro, and is a micro-framework that provides an elegant starting point for your console application. It is an unofficial and customized version of Laravel optimized for building command-line applications.
What is Laravel Dusk ?
Laravel Dusk is a powerful browser automation tool for Laravel. With Dusk you can programmatically test your own applications or visit any website on the internet using a Chrome browser. Dusk can help you to automate repetitive tasks or scrape information from other websites for example.
NB : I discovered the work of Nuno during a meetup at Algolia in November 2018, he did an incredible job for the Laravel community, follow him here: https://twitter.com/enunomaduro
Installation of Laravel Zero and Dusk
Let’s start by installing Laravel Zero by following the documentation: https://laravel-zero.com/docs/installation/
composer create-project — prefer-dist laravel-zero/laravel-zero laravel-zero-dusk
Then install Laravel Dusk:
cd laravel-zero-dusk
php laravel-zero-dusk app:install console-dusk
We will store our data in a database, in Laravel Zero the Laravel’s Eloquent component is an add-on. We can install it like that:
php laravel-zero-dusk app:install database
Remember to change the credentials of your database in the config/database.php file.
Preparation of the database
We will create 2 tables to store our data:
- twitter_account: to save the accounts handles we want to track the statistics of
- twitter_account_data: to save the account statistics of the twitter_account table
So we create 2 migrations:
php laravel-zero-dusk make:migration twitter_account_table
With this content:
<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class TwitterAccountTable extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
Schema::create('twitter_account', function (Blueprint $table) {
$table->increments('id');
$table->string('handle');
});
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
Schema::dropIfExists('twitter_account');
}
}
And:
php laravel-zero-dusk make:migration twitter_account_data_table
With this content:
<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class TwitterAccountDataTable extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
Schema::create('twitter_account_data', function (Blueprint $table) {
$table->integer('twitter_account_id')->unsigned();
$table->foreign('twitter_account_id')->references('id')->on('twitter_account')
->onUpdate('cascade')->onDelete('cascade');
$table->integer('number_follower');
$table->integer('number_following');
$table->date('date');
});
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
Schema::dropIfExists('twitter_account_data');
}
}
We then create a Seeder to add the first accounts in the twitter_account table:
php laravel-zero-dusk make:seeder TwitterAccountSeeder
With this content:
<?php
use Illuminate\Database\Seeder;
use Illuminate\Support\Facades\DB;
class TwitterAccountSeeder extends Seeder
{
/**
* Run the database seeds.
*
* @return void
*/
public function run()
{
DB::table('twitter_account')->insert(array(
array(
'handle' => 'taylorotwell'
),
array(
'handle' => 'enunomaduro'
),
array(
'handle' => 'martin_riedweg'
),
array(
'handle' => 'VitalikButerin'
),
));
}
}
We run our seeder:
php laravel-zero-dusk db:seed --class=TwitterAccountSeeder
Here we just add 4 twitter handles in our twitter_account table. You can do this with basic SQL requests or https://github.com/intonate/tinker-zero (a bridge that allows using laravel/tinker in Laravel Zero applications).
##Creation of the command that will store twitter metrics in database
Commands in Laravel Zero are explained here: https://laravel-zero.com/docs/commands/
It’s very similar to the commands of Laravel: https://laravel.com/docs/5.7/artisan#writing-commands
Create a new command named GetTwitterData:
php laravel-zero-dusk make:command GetTwitterData
Here is the code of the command (See details below):
<?php
namespace App\Commands;
use Carbon\Carbon;
use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Support\Facades\DB;
use LaravelZero\Framework\Commands\Command;
class GetTwitterData extends Command
{
/**
* The signature of the command.
*
* @var string
*/
protected $signature = 'insert:twitter_data';
/**
* The description of the command.
*
* @var string
*/
protected $description = 'Insert twitter data of account stored in the table twitter_account';
/**
* Execute the console command.
*
* @return mixed
*/
public function handle()
{
// get all twitter acount stored in database
$twitterAccounts = DB::table('twitter_account')->get();
$this->browse(function ($browser) use ($twitterAccounts) {
foreach ($twitterAccounts as $twitterAccount) {
$browser->visit('https://twitter.com/' . $twitterAccount->handle);
// get current number of followers
$numberFollower = $browser->attribute('.ProfileNav-item--followers > .ProfileNav-stat > .ProfileNav-value', 'data-count');
// get current number of following
$numberFollowing = $browser->attribute('.ProfileNav-item--following > .ProfileNav-stat > .ProfileNav-value', 'data-count');
// insert values in database
DB::table('twitter_account_data')->insert(array(
array(
'twitter_account_id' => $twitterAccount->id,
'number_follower' => $numberFollower,
'number_following' => $numberFollowing,
'date' => Carbon::now()->toDateTimeString()
),
));
}
});
}
/**
* Define the command's schedule.
*
* @param \Illuminate\Console\Scheduling\Schedule $schedule
* @return void
*/
public function schedule(Schedule $schedule)
{
$schedule->command(static::class)->everyThirtyMinutes();
}
}
So what are we doing here?
- line 35: we are just getting every handles stored in the twitter_account table, if you create and run the same Seeder as me, you should get 4 handles (@taylorotwell, @enunomaduro, @martin_riedweg and @VitalikButerin)
- line 37: for each handle, we go to the corresponding account page, for example https://twitter.com/taylorotwell for taylorotwell
- line 44 and 47: we retrieve the number of followers and following with CSS selectors: https://laravel.com/docs/master/dusk#interacting-with-elements
The exact value is in the data-count attribute
- line 50 to 57: we store the data in our twitter_account_data table
We just have to test our command:
php laravel-zero-dusk insert:twitter_data
If all goes well you should see this in your console:
Console output
You can easily automate this command using Cron: https://laravel-zero.com/docs/task-scheduling/
In the code of my GetTwitterData command I have:
public function schedule(Schedule $schedule)
{
$schedule->command(static::class)->everyThirtyMinutes();
}
So my command will run every 30 minutes ✌🏻
The code is available here: https://github.com/MartinRdw/laravel-zero-dusk
Feel free to ask me your questions/bug reports in the comment section 🙂