Here’s the last part of our series on making a Raspberry Pi Twitter bot.
Above is the picture of our completed device. When it gets new feeds, the LEGO gentleman on the sofa blinks his LED to notify you.
Today, we will set up the environment to auto post smoothly. First, let’s research the post limit for Twitter! Let’s try to avoid errors regarding things we can find out beforehand. We made it so that things get recorded in a log file so that when an error occurs during a post, we can check the details. Now that we are ready, let’s set up the auto post. We recently started using “cron”. Aside from being able to set the day and time, it also allows us to auto-execute when Raspberry Pi launches, so we gave it a shot!
We touched upon the limits in the last article and the one before that, but let’s take a closer look here (This information is as of July 2015).
Tweet posts
Definition: A Tweet may contain photos, videos, links and up to 140 characters of text.
We’re familiar with this limit. When trying to post over 140 strings through the API, we get an error saying, “Status is over 140 characters.” We can solve this with the method we covered in the last post, by considering the shortened URL to make it 140, remember?
I’m seeing an error message after posting a Tweet
- If you see a “Whoops, you already said that” error message when you post a Tweet, you’ve posted the exact same text in another Tweet recently, and should try writing something new in your next Tweet.
- You should only see this error if you post exactly the same text as a recent tweet.
As we discussed in “Making a Raspberry Pi Twitter Bot Part 1 – Twitter API“, Twitter has a function to prevent double posts. If 1 character is different, we were able to post. But when the text is exactly the same, we had to wait a few hours to post it again. When we use API, we get an error message saying, “Status is a duplicate.”
Twitter limit (API, Tweet posts, follow)
- Tweets: 2,400 per day. The daily update limit is further broken down into smaller limits for semi-hourly intervals. Retweets are counted as Tweets.
- The Tweet limit of 2,400 updates per day is further broken down into semi-hourly intervals. If you hit your account update/Tweet limit, please try again in a few hours after the limit period has elapsed.
2,400 tweets in a day! That’s more than I thought! It seems we do not have a problem. But if you want to use something that gets updated frequently like a news website RSS feed, you might want to be careful.
It seems there are no details about the “30 minute intervals” as of now (July 2015).
For more details on errors and error codes, refer to the official website.
In this case, as long as we avoid character limit and double posts, we probably won’t have a problem. But just in case they change the spec in the future, let’s make it so that the log files keep a record of your posts and errors.
We created two log files, one to record the time of execution of the last post and another to record the error logs. Below is the data that will be saved.
Execution log file (log.csv) |
RSS URL Time executed of last session |
Error log file (erroe.csv) |
RSS URL Time executed Post Error message |
Let’s manage this data in CSV format. We previously went over how to input/output CSV files.
All we had to do was customize the source of this article: “DIY Raspberry Pi Thermometer”
/var/www/news/tweet_rss.php
1 2 |
<!--?php require_once("/var/www/news/twitteroauth/autoload.php"); use AbrahamTwitterOAuthTwitterOAuth; date_default_timezone_set('Asia/Tokyo'); //Validate $consumerKey = 'c'; $consumerSecret = 'Validate'; $accessToken = 'Validate'; $accessTokenSecret = 'Validate'; $oAuth = new TwitterOAuth($consumerKey, $consumerSecret, $accessToken, $accessTokenSecret); //List of RSS $rss_list = Array(); $rss_list[] = Array('https://www.raspberrypi.org/feed/', 'RaspberryPi Update info on official website'); $rss_list[] = Array('https://www.deviceplus.com//feed/', 'DevicePlus Update info'); $rss_list[] = Array('https://www.yahoo.com/tech/rss', 'Yahoo! Tech'); //Start process for($j=0;$j<count($rss_list);$j++){ $rss_url = $rss_list[$j][0]; $pre_message = $rss_list[$j][1]; //Acquire date and time of last session $last_date = checkLog($rss_url); //Create message $rss_message = createRssMessage($rss_url, $pre_message, $last_date); //Post Tweet for($i=0;$i<count($rss_message);$i++){ tweetMessage($oAuth, $rss_url, $rss_message[$i]); } //Output performance log $log = Array($rss_url, date("Y/m/d H:i")); writeCSV($log); } return; /* Read RSS and create message for Tweet */ function createRssMessage($rss_url, $pre_message='', $last_date=NULL){ $message = array(); $xml = @simplexml_load_file($rss_url,'SimpleXMLElement',LIBXML_NOCDATA); $max_len = 140 - (mb_strlen($pre_message,'utf-8') + 23 + 2); if(isset($xml->channel)){ $channel = $xml->channel; foreach ($channel->item as $value) { //Check date if(isset($value->pubDate) && $last_date!=NULL){ $pubDate = date("Y/m/d H:i", strtotime($value->pubDate)); if($pubDate < $last_date){ continue; } } //Create main text of Tweet $str = ''; if(isset($value->title )){ $str .= $value->title."n"; } if(isset($value->description)){ $str .= strip_tags($value->description); $str = html_entity_decode($str,ENT_XHTML)."n"; } //Adjust character length if(mb_strlen($str,'utf-8')>$max_len){ $str = mb_substr($str, 0, $max_len-1, mb_detect_encoding($str)).'…'; }else{ $str = mb_substr($str, 0, $max_len, mb_detect_encoding($str)); } if(isset($value->link )){ $str = $pre_message."n".$value->link."n".$str; } $message[] = $str; } } return $message; } /* Post on Twitter */ function tweetMessage($oAuth, $rss_url, $message){ $log = Array(); //Post Tweet $response = $oAuth->post('statuses/update', array('status' => $message)); //Result if(isset($response->errors)){ //Error occured $tmp = Array(); foreach($response->errors as $err){ $tmp[] = $err->message; } $log = Array($rss_url, date("Y/m/d H:i"), $message, join($tmp, PHP_EOL)); //Output log writeErrorLog($log); }else{ //Tweet successful //Blink LED blinkLed(15,3); } return; } /* Blink LED */ function blinkLed($pin, $count){ exec('echo '.$pin.' > /sys/class/gpio/export'); exec('echo out > /sys/class/gpio/gpio'.$pin.'/direction'); exec('echo 0 > /sys/class/gpio/gpio'.$pin.'/value'); for($i=0;$i<$count;$i++){ exec('echo 1 > /sys/class/gpio/gpio'.$pin.'/value'); usleep(500000); exec('echo 0 > /sys/class/gpio/gpio'.$pin.'/value'); usleep(500000); } exec('echo '.$pin.' > /sys/class/gpio/unexport'); } /* Check date on execution log file */ function checkLog($rss_url){ $csv_file = '/var/www/news/log.csv'; $ret = date("Y/m/d H:i", strtotime("-1 day")); if(file_exists($csv_file) && ($handle = fopen($csv_file, "r")) !== FALSE) { while (($line = fgetcsv($handle, 1000, ",")) !== false) { if($line[0]==$rss_url && $line[1]!=''){ $ret = $line[1]; break; } } fclose($handle); } return $ret; } /* Write execution log file */ function writeCSV($add_data){ $csv_file = '/var/www/news/log.csv'; $csv_data = []; $flg = false; if(file_exists($csv_file) && ($handle = fopen($csv_file, "r")) !== FALSE) { while (($line = fgetcsv($handle, 1000, ",")) !== false) { if($line[0]==$add_data[0]){ $line[1] = $add_data[1]; $flg = true; } $csv_data[] = $line; } fclose($handle); } if(!$flg){ //Add $csv_data[] = $add_data; } if(($handle = fopen($csv_file, "w")) !== FALSE) { foreach ($csv_data as $data) { fputcsv( $handle, $data); } fclose($handle); } return; } /* Write error log file */ function writeErrorLog($arr){ setlocale(LC_ALL, 'ja_JP.UTF-8'); $file_path = '/var/www/news/error.csv'; if( $handle = fopen( $file_path , 'a' ) ){ fputcsv( $handle, $arr ); fclose($handle); } return; } </pre> <p>Execute command<br ?--> php/var/www/news/tweet_rss.php |
Please replace the “*****” part with the appropriate TwitterAPI key (Consumer key, Consumer secret, Access Token, Access Token Secret).
We added a simple check feature that prevents double posts based on an updated day and time.
Whether you posted a Tweet or not, the date and time will be recorded in the execution log file (log.csv) after the process.
In the next session, it will only post information that was updated after the recorded date (the day before the first time).
When an error occurs, it will be recorded in the error log file (error.csv). Just in case, we made it so that the error content will be saved.
The multiple RSS check has also been improved. The URL and the strings attached to the Tweet text will be managed in array.
Once executed, check your Twitter! If your post has been posted, it’s a success!
Figure 1
The LED blinking time is used as sleep, so it will be posted every 3-5 seconds. Check the log file accordingly.
Figure 2
This is the screen of the execution log. Are the URL and date/time recorded properly?
Figure 3
This is the error log. And here is an actual error we were able to find because of this log file (the program above has been fixed).
On the 4th row, it reads “Status is over 140 characters”. The character length is supposed to be adjusted, so why did this happen? We looked at the string that we attempted to post and pasted it into the Twitter post box, and it appears that it’s over the 140 characters limit.
Figure 4
At a closer look, “Raspi.TV” on the 3rd row is shown in blue, so it is recognized as an URL. When we post contents with two or more URLs, the 1st one gets treated as an URL and the others get counted as characters. The text was supposed to be 22 characters with the shortened URL, but instead it got counted as a normal character and resulted an the error.
Aside from this, there were many errors we noticed because of the error log file. We added the error log file because we thought it would come in handy when the API spec changes, but it already helped us in the testing stage!
Cron is our tool for Auto-execute! We already used Cron settings several times in our series. Let’s say you want to execute the command every hour. You would do the following.
crontab -e
First, bring up the cron settings screen from the LX Terminal.
00 * * * * php /var/www/news/tweet_rss.php
Then, write it like this!
Hit [Ctrl]+[O] to save, or [Ctrl]+[X] to quit settings and return to the original screen.
We used the Yahoo! “Tech” category RSS earlier, but it couldn’t keep up because it gets updated every few minutes. As a result, it disappeared from the RSS before we could Tweet it. For websites that get updated frequently like news, we may want to shorten the interval. Let’s check the overall interval before setting up the cron.
If the content gets updated only a few times a day, it worked fine with an hour interval auto-execute.
We’re done setting up the auto post! Or maybe not. Before we move on, it seems there is a way to trigger auto-execute when “Raspberry Pi launches” with cron, not just time. So we gave it a shot.
Let’s get right to the point. When dealing with “time” like this, it won’t work properly. We took a closer look to find out why. Below is the overview!
First, the program we used for the test is the following:
/var/www/news/tweet_start.php
1 2 |
<!--?php require_once("/var/www/news/twitteroauth/autoload.php"); require_once("/var/www/news/twitter_config.php"); use AbrahamTwitterOAuthTwitterOAuth; date_default_timezone_set('Asia/Tokyo'); //Acquire date $today = date("Year month date H:i"); for($i=0;$i<5;$i++){ //Create message $message = $today.' Launch Raspberry Pi!('.($i+1).')'; //Validate $consumerKey = '*****'; $consumerSecret = '*****'; $accessToken = '*****'; $accessTokenSecret = '*****'; $oAuth = new TwitterOAuth($consumerKey, $consumerSecret, $accessToken, $accessTokenSecret); //Post $response = $oAuth->post('statuses/update', array('status' => $message)); if(isset($response->errors)){ //Error occured sleep(15); }else{ //Tweet successful break; } } </pre> <p>Execute command<br ?--> php /var/www/news/tweet_start.php |
We made it so that it Tweets the time as well to avoid double posts. Also, we left room for 5 tries since it executes right after Raspberry Pi launches and there may not be any Internet connection at first. In our test, it usually posted on the 1st – 2nd try.
Now, let’s execute this program when Raspberry Pi launches.
Information on how to set up cron upon launch was found in the link on Wikipedia in the English manual page.
Crontab : Scheduling Tasks – math-linux.com
It is summarized at the bottom of this page. It seems with cron, you can either set up the date and time, or specify the strings as shown below.
There are also special strings of characters:
String | Action |
@reboot | execution at boot |
@yearly | execution once a year, “0 0 1 1 *” |
@annually | execution once a year, “0 0 1 1 *” |
@monthly | execution onnce a month, “0 0 1 * *” |
@weekly | execution once a week, “0 0 * * 0” |
@daily | execution once a day, “0 0 * * *” |
@midnight | execution once a day, “0 0 * * *” |
@hourly | execution once an hour, “0 * * * *” |
“@reboot” in the 1st row of the chart says, “execution at boost”. There are other strings to execute “once” per unit. Aside from “@reboot”, you can use numbers to express it, so you probably won’t use it too much. Ok then, let’s test it out!
Similar to before, enter “crontab-e” from the command line to bring up the cron settings screen. Then, register the command using “@reboot”.
@reboot php /var/www/news/tweet_start.php
Save the settings and reboot Raspberry Pi! Check your Twitter from a different computer when you do this.
Figure 5
If your post looks like this, you did it!
I wish I could say that, but it actually doesn’t work.
Figure 6
Here’s why. If you look closely, you will notice that the timestamp of the Tweet text is way off from the actual date and time. This is clearer when you shut down and reboot after a while. What happened was, the time when the machine shut down got posted. In my case, I booted Raspberry Pi on Monday after the holidays and noticed this. It turns out this happens because Raspberry Pi doesn’t have an internal battery. When the power goes out, the internal clock of Raspberry Pi goes too, so after rebooting, it moves based on the time when the system shut down until the clock gets reset.
I always had my LAN cable plugged in, so I didn’t notice at all. But when I shut off the network and booted it, the clock did not get reset and the system kept using the old time. From this, it seems that the “@reboot” of cron cannot keep up with the reset time. Also, in the RSS auto post tool, the posts are made based on the current time, so this will not work for us. To solve this, we can reboot the ntp, or use the ntpdate package. We gave it a try, but sometimes it was a few minutes off and sometimes it couldn’t execute in time for the “Raspberry Pi launch”. It was quite difficult to get it right.
Regardless, auto-execute upon launch is not mandatory for this tool, so I force my Raspberry Pi to “Sleep for 60 seconds, then start the process” in order to make everything work. It works without a problem with Wi-Fi, so I was able to mount the functions I wanted successfully!
By the way, if you use something called “RTC (real-time clock module”, you can prevent the clock inside Raspberry Pi from stopping.
Real-time clock
A real-time clock (RTC) is a computer clock (most often in the form of an integrated circuit) that keeps track of the current time.
Although the term often refers to the devices in personal computers, servers and embedded systems, RTCs are present in almost any electronic device which needs to keep accurate time.
We completed our Raspberry Pi Twitter Bot!
As far as auto-execute goes, there is room for improvements. But it works fine, so let’s settle for now! After all, trial and error is what makes electronic kit so much fun!
Next, we will work with a smartphone again! It’s been a while.
Before, we used the “MPD” app to listen to music. Next, we will use the “VNC” app so that you can touch the Raspberry Pi desktop screen directly!
Once you can do this, you might be able to forget about dealing with display, keyboard and mouse! They can get pretty annoying at times, right?