Category Archives: Python Lessons

How to run selenium webdriver with headless mode


Selenium is one of the powerful package available in Python , used for web automation , no doubt about that. In the last we learned how to use a web driver with webdriver manager package. In this we will learn how to use a headless mode of action in selenium webdriver.

Headless mode

What is headless mode ? In head less mode ,browser you automate work in the background and the work can be completed silently. So how to run selenium web driver in headless mode .

  • You need to create browser option and add –headless.
  • Add the option to webdriver

Chrome option

Create a chrome option as follows and add attch to the web driver.

from selenium.webdriver.chrome.options import Options
chrome_opt = Options()
chrome_opt.add_argument('--headless')
browser = webdriver.Chrome(executable_path=ChromeDriverManager().install(),chrome_options=chrome_opt)

That’s all you need to know

Fix Chrome Driver Error or Selenium webdriver error


When I started with Selenium in Python my biggest problem was the webdriver issue which I later found that there is an automatic download option in python using some special package.

The webdriver sometimes not compatible with the browser you are using. You have to manually find them from the official websites. This is messy part of selenium, I think.

We can get ride off this problem using webdriver_manager

first you need to install the Python package using pip

pip install webdriver_manager

Now you can use it in your project

from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager

driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())

The package will automatically check for compatible version and install it and no more worries about web driver and scrap the web with ease and peace.

How to pass python list to JavaScript in Flask app


You can’t use Python list directly into JavaScript, JavaScript lislt/array is not compatible with Python or wise versa. So what to do.

Solution

I have a list of names, which I want pass to a JavaScript in a Flask ninja template. Ninja template allow use to safely convert the python list to Json string then we can pass it to the JavaScript function / just JavaScript

<script type="text/javascript">

        dn = {{ distabr|tojson }}
</script>

You can’t use same route variable in multiple places in same template, it may not serve the purpose

In JavaScript Functions

Suppose you have separate JavaScript function files for some functionality like drawing analytic plots and pie diagram, or some donut chart. So how do we pass value to those functions ?

You can create a JavaScript as above, in your current template, then in function create a script variable to use it as follows.

function init_echarts() {
    var distNames=dn;
}

That’s all you need to know to embed some morris diagram to your flask app.

You should look at Morris JS , it is free and easy to use JavaScript charts library.

Pass variables to other templates in Flask


Flask is one of my Python frame work in which I build simple web application with ease and peace. Today we learn how to pass a variables to other templates in a Flask application.

Suppose your web app have a base template and a top nav bar template which included using

{% include "site_template/top_navigation.html" %}

and you want to pass a title or something else from a Index page to navigation bar where you want show up some text.

Using {% set %} you can create global variables.

The solution

We can use a set block in top of the index page as follows ,to pass variables to other sub templates

{% set mytitle=title %}

In the route it will look like

return render_template('index.html',  title=title)

and in the nav bar page we can use as mytitle as variable using

<div id="site_title"><h2  >{{ mytitle}} </h2></div>

If you had doubt and suggestion please leave a comment

RapidAPI example using request in Python


This tutorial is similar to RapidAPI example found in the official blog, with one exception, this example is users request method instead of unirest package which is not available on RapidAPI blog.

RapidAPI

RapidAPI is a website which provides useful APIs (Application Programing Interface) like Yahoo Finance, Email Verification, Geolocation which can be used to add features to your websites and apps using variety of programming languages like Python, Java, .Net etc.

Yahoo Finance example

For creating application to show stock chart using YahooFinanceAPI and Python, we are go through the following

  • Create a Rapid API Account
  • Subscribe to Yahoo Finance to get the endpoint (Which is a URL for getting information)
  • Create methods to connect API and Process data
  • Draw Plot

Install requirements

Install required packages using pip. I suggest using Python virtual environment.

pip install seaborn pandas 

Complete Source code

import json
from datetime import datetime
import matplotlib.pyplot as plt
import pandas as pd
import requests
import seaborn as sns
from matplotlib import rcParams
from past.builtins import raw_input

RAPIDAPI_KEY = "<YOUR_RAPIDAPI_KEY>"
RAPIDAPI_HOST = "<YOUR_RAPIDAPI_ENDPOINT>"

symbol_string = ""
inputdata = {}


def fetchStockData(symbol):
    url = "https://apidojo-yahoo-finance-v1.p.rapidapi.com/market/get-charts"

    querystring = {"comparisons": "%5EGDAXI%2C%5EFCHI", "region": "US", "lang": "en", "symbol": symbol,
                   "interval": "5m", "range": "1d"}

    headers = {
        'x-rapidapi-host': "apidojo-yahoo-finance-v1.p.rapidapi.com",
        'x-rapidapi-key': "275d968795msheb0c2d90f2d7a32p1141eajsnef8c254e756a"
    }
    response = requests.request("GET", url, headers=headers, params=querystring)
    if (response.status_code == 200):

        return json.loads(response.text)
    else:
        return None


def parseTimestamp(inputdata):
    timestamplist = []
    timestamplist.extend(inputdata["chart"]["result"][0]["timestamp"])
    timestamplist.extend(inputdata["chart"]["result"][0]["timestamp"])
    calendertime = []
    for ts in timestamplist:
        dt = datetime.fromtimestamp(ts)
        calendertime.append(dt.strftime("%m/%d/%Y"))
    return calendertime


def parseValues(inputdata):
    valueList = []
    valueList.extend(inputdata["chart"]["result"][0]["indicators"]["quote"][0]["open"])
    valueList.extend(inputdata["chart"]["result"][0]["indicators"]["quote"][0]["close"])
    return valueList


def attachEvents(inputdata):
    eventlist = []
    for i in range(0, len(inputdata["chart"]["result"][0]["timestamp"])):
        eventlist.append("open")
    for i in range(0, len(inputdata["chart"]["result"][0]["timestamp"])):
        eventlist.append("close")
    return eventlist


if __name__ == "__main__":
    try:
        while len(symbol_string) <= 2:
            symbol_string = raw_input("Enter the stock symbol: ")
        retdata = fetchStockData(symbol_string)

        if (None != inputdata):
            inputdata["Timestamp"] = parseTimestamp(retdata)
            inputdata["Values"] = parseValues(retdata)
            inputdata["Events"] = attachEvents(retdata)
            df = pd.DataFrame(inputdata)
            sns.set(style="darkgrid")
            rcParams['figure.figsize'] = 13, 5
            rcParams['figure.subplot.bottom'] = 0.2

            ax = sns.lineplot(x="Timestamp", y="Values", hue="Events", dashes=False, markers=True,
                              data=df, sort=False)
            ax.set_title('Symbol: ' + symbol_string)

            plt.xticks(
                rotation=45,
                horizontalalignment='right',
                fontweight='light',
                fontsize='xx-small'
            )
            plt.show()
    except Exception as e:
        print("Error")
        print(e)

Output

Streamlit – A perfect tool for data visualization


In Minimal lines of code you can visualize the data you have. Streamlit Python framework for data science and for big data visualization.

With few lines of code you can build an app with data,plots and maps instantly, which may save some work. Coupled with panda and numpy the performance goes beyond you expect.

Here is a sample code which generate sales/purchase history line chart


The streamlit app

First up all install streamlit

pip install streamlit
import streamlit as st
import pandas as pd
st.title('Sales/Purchase History')
st.text('This is a simple Streamlit app using Widgets') 
df=pd.DataFrame({'Purchase':[1200,1450,1100,12700,1340],'Sales':[12000,45000,1000,67000,3400]},index=['JAN','FEB','MAR','APR','MAY'] )
 
if st.checkbox(label='Show Frame'):
    st.write(df)
options =st.multiselect(label='Filter',options=df.columns)
 
st.line_chart(df[options])

Running

Running the streamlit app have little different than other python scripts.

 streamlit run app.py

Use always run, on browser , which enables streamlit script will automatically refreshed while you made any change in data or script.

For more about the frame work and features I suggest the official documentation

Python Library Tips: timedelta


Timedelta

Timedelta class represent not any particular date or time, but it represents a span of time. This class will be helpful when you need to run some math on date and time. Say what will be this day on after few weeks or what will be this day in future year.

Following example will make it clear

from datetime import date
from datetime import timedelta

print(f'todays date {date.today()} after 1 year will be {date.today()+ timedelta(days=365)}')

Output 
todays date 2020-06-27 after 1 year will be 2021-06-27

Here I used the f.string to construct the message , the timedelta represent one year, 365 days. You can find what will be the date before few weeks back using date -timedelta(weeks=4)

Python Library Tips: strip off letter from list


f.strings

In this series of posts , learn how to use Python Library functions

How to strip off no of letters from a list of strings

The split function of string class can be used to split words from a string. do you know how to creating a abbreviated set of strings from a list. I

With a comprehension statement in Python this can’t be hard

list=['PYTHON','CPYTHON','IPYTHON']
stripped=[ k[:3] for k in list]
print(stripped)
['PYT', 'CPY', 'IPY']

Here we strip of the each list element using : operator in the list and specify the number of characters want to keep.

You can also have the reverse order using k[3:] or can use the starting and ending position as follows

stripped=[ k[0:4] for k in list]
print(stripped)
#['PYTH', 'CPYT', 'IPYT']

stripped=[ k[1:4] for k in list]
print(stripped)
#['YTH', 'PYT', 'PYT']

Python Library Tips: f.strings


f.strings

In Python you can format string in a variety of ways.We learned how to use a %format technique in last post. In Python version 3.7 introduced f.string .

f/F strings are literals begins with letter f/F and It uses { }, curly braces for replacement of expressions with their values

spyder_version=3
python_version=2.7
print(f'I am using Spyder ID {spyder_version} with Python {python_version}')

#Output
I am using Spyder ID 3 with Python 2.7

Arbitrary expression

F"{10*10}"

#Output
100

Function call

We can also call function inside f.strings.

def multi(x,n):
    return x*n
f'{multi(2,2)}'

Multipleline

f-strings are faster than % formatting because they ares constants while f.string expressions are evaluated at runtime.

Also it is possible compose f.strings in multiple line. All strings must begin with ‘f/F’.

>>topic='panda frames'
>>lang='python'
>>version =3.8
>>say=(f'I want to use {topic}'
  f' in {lang}'
  f' version {version}')

>>say
Out[20]: 'I want to use panda frames in python version 3.8'

It can also be rewritten using ‘\’

>>say=f'I want to use {topic}' \
f' in {lang}' \
f' version {version}'

Python Library Tips: str.format


str.format()

In Python you can format string in a variety of ways.We learned how to use a %format technique in last post. In Python version 2.6 introduced str.format() function which is an improvement over % formatting.

It uses { }, curly braces for replacement fields

spyder_version=3
python_version=2.7
print('I am using Spyder ID {} with Python {}' .format(spyder_version,python_version))

#Output
I am using Spyder ID 3 with Python 2.7

You can access variable using digits starting from {0}-{n} as follows

spyder_version=3
python_version=2.7
print('I am using Python {1} and Spyder IDE {0} for data science ' .format(spyder_version,python_version))

#Output
I am using Python 2.7 and Spyder IDE 3 for data science