Python Fundamentals: Tokens.
As we are discussing Python fundamentals, so we have to consider tokens, statements, expressions, simple input, and output programs, etc. In this article, you are learning about Tokens in Python.
The smallest individual unit in a program is known as Tokens or lexical units.
Python has mainly five types of tokens which are given below:-
- Keywords
- Identifiers
- Operators
- Literals
- Punctuators
Let’s take an in-depth analysis of these five tokens that are mentioned above. As these are the most important and I should say essential elements of Python programming. So let’ s take our first sincere step towards Python programming we are initiating with the very first lexical unit that is Keywords.
1. Keywords.
Keywords in Python are the words having special meaning reserved by Python. Keywords are the reserved words, so we can not use a keyword as a variable name, function name, and identifiers. These are allotted to do a specific type of work that makes programming very easy and accessible. In Python, Keywords are case sensitive, which means you have to use that form of keywords as they are defined in its library as it treats upper and lower-case characters differently.
Keywords may vary with different versions of Python. In Python 3.8.2 total no of keywords are 35. You can get a list of keywords of your version also by typing simple code:-
import keyword
print(keyword.kwlist)
Output for list of keywords in Python.
Now you have sufficient knowledge of Keywords so we can move ahead to our second lexical unit, which is Identifiers.
2. Identifiers.
Identifiers are used as the general terminology for the names given to different parts of the Python programs. It is used to identify a function, variable, class, module, object, list, dictionaries, and other parts of programs. It also helps in differentiating between two different entities.
In Python certain fixed rules should be kept in mind before addressing identifiers that are being specified below:-
- The first character must be a letter, and the underscore (_) counts as a letter in addressing an identifier.
- An identifier must not be the keyword of Python and special character except underscore(_) (as mentioned above Python treats underscore as a letter)
- Upper and lower-case letters are treated differently. (Case sensitive)
- An identifier can’t start with a digit. For example, 1Priyam is invalid, but Priyam1 is a valid Identifier.
- Identifiers can be unlimited in length. Python has no word limit for defining an identifier.
For example, the following are some valid identifiers:-
- Pkthsource
- PK_7_9918
- PKTHSOURCE
- _Pkth
The following are some invalid identifiers:-
- Priyam.katiyar (contains special character dot)
- try (reserved keyword)
- 7Pkth(Starting with a digit)
I hope you all have understood all concepts of Keywords and Identifiers. Trust me; this much information is more than enough. So now, we can move forward to our third lexical unit, which is Operator in Python.
3. Operators.
Python Operators are tokens/lexical unit that trigger some computation/action when applied to variables and other objects in an expression. Variables and objects to which the computations applied called operands. So, the implementation of operator requires some operands to work.
Operators in Python are divided into eight groups which are given below with some brief description:-
- Arithmetic Operators
- Shift Operators
- Bitwise operators
- Identity Operators
- Relational Operators
- Logical Operators
- Membership Operators
- Assignment Operators
These operators are assigned to perform mathematical operations. It mainly used with numeric values.
List of arithmetic operators are given below:-
- (+) Addition
- (-) Subtraction
- (/) Division
- (*) Multiplication
- (//) Floor division
- (%) Modulus
- (**) Exponential
Shift Operators.
Shift operators are used to shifting the content. There are only two shift operators in Python.
- (<<) Shift left
- (>>) Shift right
Bitwise operators.
Bitwise operators are used to comparing two operands. These operators treat operands as binary digits as their name.All three bitwise operators that exist in Python are given below:-
- (&) Bitwise AND
- (|) Bitwise OR
- (^) Bitwise XOR
Identity Operators.
Identity Operators are used to comparing the objects. They mainly check if two values (or variables) are located on the same part of the memory or not. There are only two Identity operators who are given below:-
- (is) True if the operands are identical.
- (is not) True if the operands are not identical.
Relational Operators.
Relational operators are used to comparing two operands and check whether given relation exists or not. List of relational operators are given below:-
- (<) Less than.
- (>) Grater than.
- (<=) Less than or equal to.
- (>=) Greater than or equal to.
- (==) Equal to.
- (!=) Not equal to.
Logical Operators.
Logical operators are used to combining conditional statements. All three logical operators are given below:-
- (and) Returns true if both the operands are true.
- (or) Returns true if one of the operand is true.
- (not) Returns true if the operand is false.
Membership Operators.
Membership operators are used to testing whether a value or variable is found in a given sequence. They are mainly used for list, tuple, dictionary, set and string. There are two membership operators which are given below:-
- (in) To check whether a variable is in sequence.
- (not in) To check whether a variable is not in sequence.
Assignment Operators.
Assignment operators are used to assigning value to the variable. Some arithmetic operators also linked with assignment operators whose functionality is as like arithmetic operators, but the only difference is there it also assigns value to its counter variable. List of all assignment operators are provided below:-
- (=) Assign value to its counter variable.
- (/=) Assign quotient after calculating to its counter variable.
- (+=) To assign sum after calculating to its counter variable.
- (*=) Assign product after calculating to its counter variable.
- (%=) To assign remainder after calculating to its counter variable.
- (-=) Assign difference after calculating to its counter variable.
- (**=) To assign product after calculating to its counter variable.
- (//=) Assign floor division after calculating to its counter variable.
I think this much information is more than enough for Operators in Python. I hope you all have understood Operators along with all it’s eight types. So now, we can move towards our forth lexical unit/token, which is Literals/Values in Python.
4. Literals/Values in Python.
Literals are data items that have a fixed value. It can be the string, float, integer, list, tuple and dictionary types. Python allows several kinds of literals:-
- String literals.
- Numeric literals.
- Boolean literals.
- Special literal(None)
Brief description of these literals:-
String literals.
In Python, you can form string literals by enclosing text in both forms of quotes(single quotes or double quotes). For example, “Priyam“, ‘priyam1122‘, “pkthsource-no-1” are all string literals in Python.
Numeric literals.
The numeric literals in Python can belong to integers, floats or complex numbers. So let’s see how to differentiate and implement it in Python.
- Integers are positive or negative whole numbers with no decimals and are represented by (int) tag in Python.
- Floats are written with a decimal point dividing the integer and fractional parts. These are represented by (float) tag in Python.
- Complex numbers are in the form of a+bj, where a and b are floats. These are represented by (complex) tag in Python.
Boolean literals.
A Boolean literal in Python is used to represent Boolean values. True and False are only two Boolean literals in Python. These can either have value as True or False.
Special literal (None).
None literal represents the absence of a value. Generally, it is used to indicate the end of lists in Python. The None value means “There’s nothing here“.
We have covered Literals so, we can move forward toward our last lexical unit which is Punctuators in Python.
5. Punctuators.
Punctuators are symbols that are used in programming languages to organize programming-sentence, structures, and indicate the rhythm and emphasis of expressions, statements and program structure. A punctuator can also be a token that is used in the syntax of the pre processor.
Conclusion.
In this session we have discussed all about tokens in Python i.e. Keywords, Identifiers, Operators, Literals, Punctuators and its subtopics i.e. Arithmetic Operators, Shift Operators, Bitwise operators, Identity Operators, Relational Operators, Logical Operators, Membership Operators, Assignment Operators, String literals, Numeric literals, Boolean literals, Special literal(None) and many more. I hope you have found this article helpful and your all doubts are clear. If you have any question or suggestion then let me know in the comment section trust me your opinions means a lot for us.