Taiwan Stock Tape Reading Periodically Using Web Scraping Technology with GUI

Abstract

Stock tape reading involves surveilling stock prices once in a while and recording stock prices. The method of observing stock prices may be television or stock exchange. The time step for recoding stock prices is every stock user’s experience and their theory, perhaps 3 min or 2 h and so on. As an example, the Taiwan stock market starts at 9:00 a.m. to 13:30 p.m. It will have a 4 h operating time. Splitting the 4 h operating time for tape reading is the skill of stock users. The stock price sequence generated by tape reading can be predicted by stock users, but finally, it is the stock user’s experience. Therefore, the meaning of tape reading is to record the stock price, but its concept should have no prediction purpose. This study used thread technology and proposed a tape-reading method with web scraping. This method can periodically scrape stock prices and generate a stock price sequence to Excel file. This application can satisfy the demand of these stock users, who are called day trading users. Because these day trading users want to gain stock price sequences minute by minute, rather than the stock exchange format day by day, and also ones which are better than the those provided by the stock website service, because its stock sequence format is limited and not normalized, these day trading users think that minute-by-minute stock price sequences are very clear to forecast. This study implemented the prior scheme and designed the GUI to query a company’s stock price and its stock news, even per second, etc., and how long it took to update the stock price, and the GUI also included a time-up feature to stop scraping stock prices if users just wanted to scrape stock prices periodically.

1. Introduction

In the Taiwan stock market, there is a jargon term called tape reading, and its service user surface is for day trading users. Day trading is a strategy for short-term investment, that is, to gain the stock benefit with stock price float-up and -down trends on the same day, but not to care for market development. The definition of day trading is not equal to that of traditional exchange. Day trading users and these concepts focus on stock price behavior in one day, not the company’s ability to influence long-term corporate stock prices. Thus, tape-reading users require numerous stock sequences that are of a better type, such as the Excel format, rather than the current stock service website, such as Yahoo and Anue’s limited reload stock sequence and stock fundamental figure. The challenge of this application is that the reload period of stock prices is limited in Yahoo and Anue and these stock platform websites only reload at 9:00 a.m. on that day. The time stamp of stock prices in Yahoo or Anue is not flexible. Thus, this study will deal with this challenge, as this study application can freely select updating stock price time for 30 min, solve the nonflexible stock price time stamp problem in Yahoo and Anue, and record stock prices using this proposed method to face the difficulty of limited stock prices reloading. When designing such a system as this, we will face the technical challenge of establishing so many tabs to scrape enormous amounts of stock price data that the user wants, and a problem as in the recycle program application may occur; thus, it must be used for an object concept to create tab instances to effectively and robustly trawl the stock price. To compare with existing approaches, the proposed method adds the feature of sending stock sequences back to the user’s email, which is better that the existing Yahoo and Anue services, and it provides a more flexible stock timestamp. There are many day-trading users in Taiwan who want to use this tape-reading method to earn stock price rewards on the day in a short schedule. The tape reading surveils the stock price of some companies that have stock ticker symbols, and then records the stock price in a possible note method. In addition, the observing method of noting stock prices may be television or stock exchange. The interval time step of recording stock price varies according to every stock user’s experience and their theory may require them to record every 2 min, 3 min, 10 min, half an hour, 2 h, etc. As an example, the Taiwan stock market starts at 9:00 a.m. and closes at 13:30 p.m. It has a 4 h operating time if it contains board lot time after 13:00 p.m. Splitting the 4 h operating time for tape reading is the skill of stock users. The stock price sequence generated by tape reading can be predicted by stock users, such as tape-reading users, a concept similar to that of Tiwari et al. [1]. However, it is the stock user’s experience. This study proposed a tape-reading method with web scraping using a thread structure. This method can periodically scrape the stock price of a company that the user wants to query and must have a format stock ticker symbol; this method then generates the company’s stock price sequence in an Excel format. This application can satisfy the demands of some stock users, who are called tape-reading users. Because these tape-reading users want to gain stock price sequences minute by minute, rather than the stock exchange format of day by day, these users think that the minute-by-minute stock price sequence is very clear to forecast and also better than Anue or Yahoo and other stock search websites’ stock sequence, which have no regular minute frequency. The proposed method implemented a prior scheme and designed the GUI to search and query companies’ stock prices. In addition, the proposed method can also scrape the stock news of some companies as an extra feature while simultaneously querying the stock price sequence of these companies. It can simulate these stock users’ searches and obtain some of the companies’ stock, and a bonus gain is the stock news of these companies; and according to the stock news regarding the good or bad information about these reflected companies, such as Zhao et al. [2]; stock users can use this gained stock news as economic suggestions to forecast the stock price up and down movements of these companies. These prior scenes can simulate and modify the application scheme. The GUI that this research designed can even set a per-second setting, and so on, how long between stock price updates, and a time-up feature to stop scraping stock prices if the user just wants to scrape stock prices periodically. This research method additionally designed and added email-sending features. This feature simulates stock users finishing scraping stock prices and tape reading to transfer stock price sequences as Excel files, and they can send stock price sequences in Excel format to their email as backup. In this email-sending application scene, some stock users in the other place are strangers on the device, and then stock users use the stock scraping with a tape reading service proposed in this research. However, the stock price sequence generated by this service will save the device as an Excel file. If it can send a stock price Excel file to stock the user’s email, then it is a system such as the concept of cloud backup checking. This research will provide a solution so that stock users can freely and selectively send the generated price sequence back to their email. As such, it is convenient.

Figure 1 and Figure 2 are the application GUI of the proposed method that is implemented and designed. The GUI tab shown in Figure 1 is called manual mode. The GUI tab shown in Figure 2 is called timing mode. The manual mode only texts into the stock ticker symbol, updates the per second setting, and it allows the user to change the Excel file name and email, and then the manual mode tab will display the stock price of these companies. The timing mode texts into extra timing and prior variables, and then shows the stock information, as shown in Figure 2.

The previously implemented GUI is introduced first and shows some stock price experiment of tape reading with web scraping. The remainder of this paper is organized as follows: Section 2 describes some related work on stock price web scraping and other methods. Section 3 introduces a web scraping scheme. Section 4 provides the experimental results of tape reading with stock-price scraping. Section 5 concludes the proposed method.

2. Related Work

In the region of web scraping and data crawling domain, Goel et al. [3] used the Python programming language to design a web content mining application that scrapes enormous amounts of information regarding colleges and courses to provide subject selection. In addition, in another study, data was displayed for the data type domain of prior colleges, where Nurtanio et al. [4] used a web scraping solution to crawl images on the Bing search engine and download them back to a local device. Nekrasov et al. [5] detected geospatial map information on a web page as a web data scraping technique. Rege et al. [6] performed web data mining of jobs to discover and analyze job market trends. Vela et al. [7] scraped public transport data such as the city bus or intercity bus information on the Internet to create related applications. Murali [8] used a data mining method to scrape the price of fruits and vegetables on an online commerce website, sorted the product price to clear, and then established it to the dictionary for an advanced process, such as ranking the product price from high to low. Diouf et al. [9] used different web scraping tools to crawl numerous website data. Singrodia et al. [10] introduced the concept of web scraping from easy to hard. Uzun [11] presented the concept of a document object model tree to scrape website data. Pujari et al. [12] selected the XAMPP platform as the display surface to implement a web scraping scene application. Tiwari et al. [1] used the web scraping technique to catch stock price data and predicted the stock price trend; in addition, they compared the predicted results with traditional statistics and neural network forecast method usage. Zhao et al. [2] considered and discussed the relationship between stock news and stock price trends. Kanade et al. [13] applied a web scraping scene to stock price crawling. This related method scrapes the Yahoo news data related to stock price, and also the Twitter users’ sentiment regarding stock and Yahoo stock price data regarding the company. These prior methods are organized as

Table 1. Comparative related study.

No.	Authors	Application of Web Scraping or Purpose
1	Goel et al. [3]	Colleges and courses
2	Nurtanio et al. [4]	Image on the Bing search engine
3	Nekrasov et al. [5]	Geospatial map
4	Rege et al. [6]	Job market trend
5	Vela et al. [7]	Public transport
6	Ranjani Murali [8]	Product price on commerce website
7	Tiwari et al. [1]	Stock price and prediction
8	Zhao et al. [2]	Stock news and stock price
9	Kanade et al. [13]	Twitter sentiment and stock information
10	Diouf et al. [9]	Reviewable trying many tools to scrape many website
11	Singrodia et al. [10]	Concept introduction from easy to hard
12	E. Uzun [11]	Using document object model tree concept to scrape
13	Pujari et al. [12]	Applying XAMPP to scrape

.

According to prior related methods, the application domain of some methods is mapped to the stock price sequences in this study. There are some methods’ solution scenes which focus on another region, such as education, online commerce, and photos. However, some related methods have no specific purpose to deal with problems and demands, and show the summary or review of web scraping, or concept introduction regarding scraping data and web crawling. Pujari et al. [12] and Uzun [11] used techniques such as XAMPP and document object model tree to scrape web data; however, they did not deal with a specific region’s demand.

With the prior summary comparing the method table, No. 1 to No. 6 methods first display that the web scraping technique can deal with an enormous region and its demand, and No. 7 to No. 9 solutions focus on the same topic as in this study, which is also a stock domain. Other related methods are reviewable experiments or theory introductions regarding web scraping; otherwise, they use their solution to implement data mining on web tasks. This study aims to compare with the No. 7 to No. 9. methods and continue the discussion. In

Table 2. Stock scraping comparative related work.

No.	Authors	Application of Web Scraping or Purpose
1	Proposed method	Stock price, news, trend GUI
2	Tiwari et al. [1]	Stock price and predict
3	Zhao et al. [2]	Stock news and stock price
4	Kanade et al. [13]	Twitter sentiment and stock information

, Kanade et al. [13] used Twitter sentiment to help with stock analysis, which encourages the proposed method to further classify sentiment to stock trends in an advanced study, but not the presented mission in the proposed method. Zhao et al. [2] applied stock news as a formula and theory to guess the stock price up and down movements; the proposed method also, as a bonus, scrapes and gains stock news to follow the method of Zhao et al. [2]. The Tiwari et al. [1] method or all related studies guide the proposed method to first crawl stock information to display and then provide it as a service for tape-reading users.

As shown in

Table 3. Stock-scraping comparative stock service websites.

No.	Application	Technology of Web Scraping or Purpose
1	Proposed method	Thread algorithm
2	Anue	ASP, AJAX
3	Yahoo	Real time—gets the stock price from physical exchange

, in recent times, the technique of web scraping and data crawling domain, famous studies such as the Yahoo stock service web page and Anue use ASP or AJAX methods to mine stock price information from the stock exchange.

3. Proposed Method

The proposed method uses the Python programming language to implement the proposed scheme and design it to the application GUI. The operation step is as follows: First, the urlopen() function is used to obtain the original context of the website and operate beautiful soup toolkit to add the original data to useful information. For example, the original context is so many div and span elements; there are some effective texts in these elements, perhaps related to this website’s topic and issue. Perhaps this website’s topic is computer shopping; then, there are some computer brands, their price, and the computer performance in sections, such as div and span blocks, which can be found and sorted to the clear list, making it another interesting application.

The flowchart in Figure 3 was designed and proposed in this study, and the prior concept all as Figure 4.

The urlopen() function is a request concept. First, the user infosurfs the browser, representing the client side, to post the request to the server, as shown in step 1 in Figure 5. In step 2, shown in Figure 5, the server side receives a request based on its content for processing and response. It will respond to the website format, which contains the html tag, after the browser receives the html tag website format to render and display for the user. All the examples are shown in Figure 5.

Therefore, according to prior introduction, what the web scraping method has to do is to catch the response data as fast as possible before the server receives a response and sends it to the client browser. The urlopen() function that processes this task first simulates the user’s browser and then sends the request to the server. When the server deals with prior sending, the urlopen() function must try to be ahead of the web operation response time for the catch response data. These data are the effective information of the website.

After the urlopen() function obtains effective data from the web page, it has to use the beautiful soup [14] toolkit for additional useful information, as shown in Figure 6 and Figure 7. mining the stock news of a company. As an example of scraping stock news that this study wants to implement, if the company’s stock news is queried, there are so many li blocks on the HTML web page. The stock news title is hidden in the li element and related to a higher level object in HTML. Using a beautiful soup function and obtaining text features can provide a plain context that can be queried. Applying the panda series and data frame concept of Python can sort these contexts into clear formats, such as sheets, and display on the GUI.

With the example of crawling stock prices, there are two possible website sources. The first is website 1, the stock service website called Anue, and another, website 2, is the Yahoo stock website in Taiwan. The proposed method will crawl the company name, stock price of the company, stock trends, and stock price fluctuations. However, the problem of the Yahoo stock website is that stock price fluctuation data are not related to any span block or element. Scraping stock information from Yahoo is difficult, and the request limit of the programmable scraping of the Yahoo stock website is low. The prior concept will transfer to display in Algorithm 1.

Algorithm 1 Thread Stock Web Scraping

Input: stock attribute that want to query company

Output: stock price sequence with regular time interval frequency

def switch ():

global flag

flag = Truestart ()

def start ():

def run ():

while (flag == True)

stock company, stock price and so on variable link into stock service website.

While (flag == True)

record the stock price, stock trend, stock company, regular date time, and append them into stock sequence data frame.

time sleep (update interval time)

if (flag == False)

stop stock scraping and generate to stock sequence excel.

create thread target == run ()

thread execute

def stop ():

global flag

flag = False

start button command = switch ()

stop button command = stop ()

When the prior tools, regarding the beautiful soup, that extract website content and the request urlopen() function are introduced, completed and otherwise, the process successfully crawls the stock data. However, there is a problem. If the GUI is used to display the stock scraping results, there is an issue that must be solved. Because the Python GUI regarding the tkinter is a single thread that only maintains the GUI window, if it creates another new task, such as clicking the button to scrape the stock price, it will bunch to the original thread mission that holds on the GUI surface. Therefore, the solution to deal with it is to create a new thread to take the target of the start button function, such as a stock-tape-reading event, and also design the stop button to break out of the scraping task; or, an alternative solution is a stock-crawling endless loop that cannot stop the attack on web scraping source website. These flow chart concepts are shown in Figure 8. and try again demonstrate the input and output with the flow chart type as Figure 9.

4. Experimental Results

The experimental results are divided into two parts as follows: the stock price sequence scraped manually as Figure 10, and the scraped stock price transferred to the Excel file generated by the timing mode as Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15.

As the prior experimental results demonstrate, the operation step of the application GUI is the first entry stock ticker symbol that is to be queried, and the number of seconds to update and show the stock price and stock trend. Otherwise, the generated stock Excel just as Figure 16. And the Excel file name that it wants to call, and the selectivity email when the tape is read, are sent to the user’s email. In addition, the timing mode has the same meaning as the prior manual mode, and the execution time of tape reading and the delay time of tape reading postpone the start. The proposed stock tape reading based on a web scraping method has been tested on a computer platform specific to a CPU that can handle this scraping application; it can be fluently processed on the following CPUs: Intel Celeron G 1620, i5-4570, i5-9400, i7-9700, i7-10700, i9-9900; these are modern operation environments, and, furthermore, can execute on old system with a prior generation CPU that completely finish the stock crawling mission, and more experimental data as

Table 4. Ticker symbol stock price sample table.

Stock Ticker Symbol	Maximum Sampling Period	Stock Price Update Gap Time	Stock Price Change	Start Query Time	Response Time
3707 Episil	1884 s	7 s	6	0.9 s	0.7 s
3017 AVC	1883 s	9 s	10	1.2 s	0.9 s
2374 Ability enter	1374 s	7 s	6	0.5 s	0.7 s

.

And there are the important thing to evaluate the real stock price sequence between the stock platform website resource with these stock data and the stock price sequence Excel file generated by this propose application, all prior work as Figure 17.

This research method survey the user opinion, and collect the user concept data for furthermore process the evaluation as Figure 18. to enhance the useful of this application designed by this proposed scheme.

5. Discussion

The application GUI that is proposed by this study demonstrates a successful rate of completely scraping the stock data; it is influenced by crawling the stock information, similar to stock price, stock trend, and the stock ticker symbol; otherwise, the company name reflects the query corporation, but the stock platform website flow that is busy will obtain the not-entity object, meaning it failed to scrape the stock data. This proposed method will influence stock market applications and the behavior of stock users regarding faster trading, because of some users’ surveys processed by this research method. It found that there is a huge number of users who would like to use the application designed by this research, because this implementation quickly queries stock prices and automatically transfers the data to the record sequence, thus it can help people effectively get the stock data and gradually turn them to the short term trading ground. Additionally, this proposed schema has solved the problem about users surveilling the stock price just with the pure stock information platform website, and just to be the long-term investment stock users to passively wait for stock price change and except so stock earn, this application try to let stock users handle more real-time stock prices and more positively control the stock data flow to equal their trading schedule. This implementation application is suitable for the utf-8-based system environment, but it may appear as a language and time zone problem in other countries and devices.

6. Conclusions

With the introduction of the concept and the proposed scheme, the programming operation can obtain effective experimental results. It is a possible solution that is proposed in this study, and then in advanced works, but actually not implemented in the present state; this study will further research thread technology to process the tape reading on stock web scraping to become more robust, and attempt to analyze the stock news via the implemented GUI displays in this study. It can use neural network technology to transfer the stock news sequence text to normalization neurons, and through the hidden layer in the neural network to propagate sentiment—positive or negative classification of the stock news regarding the company. In addition, advising the user of a stock price trend of the company with stock news, extracting the stock news keywords or, more effectively, phrases regarding these companies into text summaries using neural networks and machine learning algorithms.