Information Theory and Digital Communications

Home Admissions Course Guide Information Theory and Digital Communications

This Course Guide has been taken from the most recent presentation of the course. It would be useful for reference purposes but please note that there may be updates for the following presentation.

ELEC S323

Information Theory and Digital Communications

Welcome to ELEC S323 Information Theory and Digital Communications.

This is a two-semester, ten-credit course for Hong Kong Metropolitan University (HKMU) students seeking the BSc(Hons) in Communications Technology. It is also an optional higher level course of BSc(Hons) in Electronics. The study units, supplementary readings, and activities and self-tests will help you master the topics over a period of around 36 weeks.

 

About the course

The central topics of the course are fundamental limits in information theory and reliable transmission of digital signals. Four major themes are covered in this course: 1) The overall architecture of digital telecommunication systems; 2) The fundamental limits of transmitting information reliably under noisy environment; 3) The design and analysis techniques to handle the random signals, of the impairments suffered by signals during transmission; 4) The various techniques to minimize the occurrence of errors resulting from noise and distortion during signal transmission.

 

Purpose of this Course Guide

You have probably taken other courses at HKMU, so you should now be well aware of both the study skills required for distance learning and how HKMU courses are organized. It is still recommended, though, that you read this Course Guide thoroughly before looking at the study units or supplementary readings.

The Course Guide tells you briefly what the course is about and how you can work your way through the material. It suggests the amount of time you are likely to need to spend in order to complete the course and will give you a general idea when your tutor-marked assignments are due. For detailed information on assignments, however, please refer to the assignment File and, for information on due dates and cut off times for work to be submitted, please refer to the Presentation Schedule.

In distance learning, as practised by HKMU, the study units (not the tutor) replace the university lecturer. This is one of the great advantages of distance learning: you can read and work through specially designed study material at your own pace at times and places that suit you best. Think of it as reading the lecture instead of hearing it from a lecturer. In the same way that a lecturer might set you some reading to do, the study unit will tell you when to read your textbook or other material. In the same way that a lecturer might give you an in-class exercise, your study unit will have exercises (called by various names) for you to do at appropriate points. You are also likely to find review questions at the end of each unit. Do them all, as these exercises and questions give you the practice necessary to achieve the objectives of the course and to pass the examination.

The course aims to:

  1. To develop learners' knowledge in the fundamental limits of information theory.
  2. To provide students with a basic knowledge of the major techniques and types of system used in digital communications, and an understanding of the principles on which they are based.
  3. To enable students to follow and interpret descriptions of systems which are new to them, and which have not been covered specifically in the course.
  4. To develop students' capability to evaluate different digital system designs and propose a system appropriate for a specified application.

After completing the course the learners should be able to:

  1. explain the specialized terminology in information theory and digital communications.
  2. perform simple reliability study and queuing analysis of a communication system.
  3. describe and explain the basic operations of digital signal transmission in the presence of noise.
  4. propose a coding or modulation scheme for a given channel and intended traffic.
  5. recognize when a proposed system is unrealistic or failed to satisfy a specification.

Course organization

The following chart gives a general overview of the course structure.

 

UnitTitleWeeksAssessment activity
(end of unit)
1Introduction1 
2Reliability, Traffic and Information7 
3Digital Signals3Assignment 1
4Noise3 
5Coding and Modulation5Assignment 2
6Comparison of digital modulation systems3 
7Channel Coding4Assignment 3
8Source coding and channel capacity3 
9Equalization2 
10Spread spectrum communication3Assignment 4
 Revision2 
 TOTAL36 

 

Unit 1 Introduction

Contents

Introduction

  • Historical Review

Why go digital?

  • Advantages of digital communications

Block diagram of a typical digital communication system

Structure of the course

 

Unit 2 Reliability, Traffic and Information

Contents

Introduction

Probability

  • relative frequencies and probabilities
  • Venn diagram
  • conditional probabilities

Reliability

  • basic concepts
  • failure rate
  • exponential distribution
  • average lifetimes
  • practical applications of failure rate
  • reliability of systems and systems element

Binomial, Poisson and normal distribution

  • Binomial distribution
  • Poisson distribution
  • normal distribution

Traffic

  • blocking probability and grade of service
  • queues
  • the M/M/1 queue
  • multiple server queue

Coding and Information

  • source coding
  • quantifying information
  • channel coding: error detection and correction
  • cyclic redundancy checks
  • channel capacity

Unit 3 Digital Signals

Contents

Introduction

Systems and signals

  • linear models
  • Fourier analysis
  • bilateral spectra
  • amplitude distortion and phase distortion

Modeling individual pulse

  • the spectrum of an isolated, rectangular pulse
  • input-output relationships
  • other pulse shapes

Aspects of digital signal transmission

  • threshold detection
  • intersymbol interference
  • coding aspects

Pulse code modulation

  • sampling
  • encoding and decoding
  • signal recovery

Systems aspects of pulse code modulation

  • time division multiplexing
  • regenerative repeaters
  • codecs

Unit 4 Noise

Contents

Introduction

sources of noise
strategies of optimizing noise performance
structure of the block

Characterizing random noise

time domain measures of a random variable
frequency domain characterization of a random variable
quantisation noise revisited

Sources of noise and distortion

thermal (Johnson) noise
shot noise
1/f noise
other causes of signal degradation

Noise in circuits and systems

basic concepts
noise temperature
noise factor
noise factor or noise temperature

Noise in digital systems

threshold detection and the Gaussian distribution
receiver filtering

Unit 5 Coding and Modulation

Contents

Introduction

Line codes

  • code radix, redundancy and efficiency
  • interface codes
  • block codes

Codes for line systems

  • metallic line systems
  • optical fibre line systems
  • trade-offs in multi-level coding

Decoding, errors and timing

  • error detection
  • the decoding of data corrupted by errors
  • timing information and synchronization
  • increasing the timing content of codes
  • block alignment

The spectral characteristics of coded data

  • code spectrum
  • low-frequency spectral components
  • high-frequency spectral components

Differential coding and scrambling

  • non-redundant coding
  • differential coding
  • scrambling

Convolutional coding

  • encoding
  • decoding and error detection
  • practical convolutional codes

Modulation

  • the modulation of digital data
  • spectra
  • noise immunity
  • comparison of binary OOK, PSK and FSK
  • multi-level modulation schemes

Demodulation methods

  • envelope demodulation
  • homodyne demodulation
  • carrier extraction for homodyne detection

Modems

  • modulation schemes
  • bit rates
  • equalization
  • duplex operation
  • trellis code modulation

Matched filter detection

  • baseband systems
  • modulated systems

Practical baseband receivers

  • intersymbol interference
  • equalization
  • pulse shape
  • crosstalk
  • cost

Unit 6 Comparison of digital modulation systems

Contents

Introduction

Error rates as a function of signal-to-noise (S/N) ratio per bit

  • Binary modulation scheme
  • Multi-level modulation

Bandwidth efficiency for digital modulation schemes

  • Multi-level ASK or PAM
  • QAM
  • FSK
  • Channel capacity
  • Comparison of digital modulation schemes

Unit 7 Channel Coding

Contents

Introduction

Linear Block Codes

  • Code linearity
  • Minimum distance and error-correcting capabilities
  • Coding gain
  • Generator matrix
  • Systematic codes
  • Parity check matrix
  • Syndrome
  • Syndrome decoding

Cyclic codes

  • Generator polynomial
  • Error detection
  • Well-known block codes
  • Performance of block codes

Soft-decision Decoding vs Hard-decision Decoding

Convolutional Codes

  • Encoding of convolution code
  • Decoding of convolutional code
  • Properties of convolutional code
  • Performance of convolutional code

Tellis-coded Modulation

Interleaving

  • Block interleaver
  • Convolutional interleaver

Concatenated block code

 

Unit 8 Source Coding and Channel Capacity

Contents

Introduction

Measure of Information

  • Information source
  • Entropy

Source coding

  • Source coding theorem
  • Classes of code
  • Kraft-McMillan inequality
  • Huffman coding
  • Lempel-Ziv coding

Discrete memoryless channels

  • Noiseless channel
  • Binary symmetric channel

Channel capacity

  • Joint and conditional entropy
  • Mutual information
  • Channel capacity
  • Channel coding theorem
  • Differential entropy

Additive white Gaussian noise channel

  • Information capacity theorem
  • Implication of information capacity theorem

Rate distortion theory

 

Unit 9 Equalization

Contents

Introduction

  • Linear channel and receiver
  • Noise and distortion in the channel leading to ISI

Linear equalizer

  • Zero-forcing equalizer
  • Effect of noise
  • Transversal filter
  • Noise enhancement
  • Minimum mean-square error criterion

Adaptive equalizer

  • Method of the steepest descent
  • Decision feedback equalizer

Unit 10 Spread spectrum communication

Contents

Multiplexing and multiple access

  • Different types of multiple access
  • Spread spectrum

Pseudo random sequences

  • Properties of pseudo random sequences

Direct-sequence spread spectrum

  • Chip rate
  • Bandwidth requirement for the DS signal
  • Spreading effect at the receiver
  • Processing gain

Frequency hopping

  • Relationship between hop rate and symbol rate
  • Processing gain

Comparison between DS and FH

Study units

You should read the study units carefully as they can guide your learning and tell you how to approach any assignment related to the unit. Otherwise, you may miss important information. You must read the study units and you should also read all suggested supplementary readings. They are not alternatives. Moreover, you should also read articles in newspapers and journals and other books related to the topics. The more widely you read, the better your appreciation and understanding of the course.

Each unit is divided into a number of sections. The first section provides the objectives of that unit and introduces the materials to be covered. The next section constitutes the contents of the study unit. This section will guide your learning and direct you to complete the activities and self-tests.

There are 10 study units in Information Theory and Digital Communications. Each study unit consists of one to eight weeks' work and includes specific objectives, directions for study, commentaries on readings, additional materials, and summaries of key issues and ideas. The units direct learners to work on exercises related to the required readings and provide practice exercises and self tests, where appropriate. In general, these exercises question learners on the material they have just covered or require them to apply it in some way and, thereby, help learners to assess their progress and to reinforce their understanding of the material. Together with tutor-marked assignments, these exercises will assist learners in achieving the stated learning objectives of the individual units and of the course.

 

References:

The following books can be used as references to provide further information and they may be helpful for your studies in ELEC S323. However, you are NOT required to buy them.

  • Simon Haykin, Communication systems, 5/e, Wiley, 2009.
  • Bernard Sklar, Digital Communications Fundamentals and Applications, 2/e, Prentice Hall, 2001.

Equipment required by students and tutors

The minimum configuration of the computer system is:

  • Pentium 4 CPU
  • Microsoft Windows XP
  • VGA display card & Colour monitor
  • 256 MB RAM
  • 2 GB free space hard disk
  • CD-ROM Drive (8X or better), sound card & speaker
  • 56 kbps modem.
  • Mouse + printer

The assessment of this course consists of a continuous assessment component and a final examination. Your progress throughout the course (the continuous assessment component) will be assessed through four assignments. All the 4 assignments will counted for assessment. The assignments are designed to test the students on practical work and analysis skills. At the end of the course you will be required to sit a three-hour examination.

Your overall course mark will be calculated from the results of your assignments and from your examination result as follows:

4 assignments (all equally weighted) 50%
Final Examination 50%
TOTAL 100%

The final examination is a written paper of three hours, and you will attempt the questions without the help of any notes or printed materials relating to the course. A simple scientific calculator is allowed. You will be sent a Specimen Examination Paper, which resembles the actual paper in both style and format, so that you can get some idea of what to expect.

The course grade is mainly determined by the overall course score (CS) yet students are normally required to obtain a minimum in both overall examination score (OES) and overall continuous assessment score (OCAS) set by the University in order to obtain a Pass result. To be awarded a particular course grade, student must meet the minimum CS set by the Award Committee.

 

Assignments

This course is designed to help you move easily from the required readings to the assignments and examination. You are expected to apply information and techniques presented during the course when completing the assignments.

You must submit assignments to your tutor for formal assessment in accordance with the due dates stated on each assignment. The due dates can also be found on the Presentation Schedule. The self-tests or SAQs are, by definition, not part of your formal assessment, but it is very important that you complete them as you work through the units. They not only expose you to the types of problem you are required to complete for the tutor-marked assignments, but they also reflect the demands of the unit objectives and are designed to help you understand and apply the principles covered in the units.

 

E-submission of assignments

There are four assignments for this course. These are all assignment exercises. You are required to submit your assignments via the e-submission in the OLE. You must submit the assignments on or before the corresponding due date. You can prepare your assignments using word processing software (e.g. Word) and then upload the pdf file to OLE. Or you can complete your assignments on paper and convert them to soft copy by scanning or taking pictures. The recommended format is pdf, jpeg or png (the file size should be less than 10 Mb). You will be able to complete all forms of assessment from the information and material contained in your study guides; however, it is preferable in all degree-level education to demonstrate that you have read and researched more widely than the required minimum. Using other references gives you a slightly different viewpoint and may give you a deeper understanding of the subject.

Your tutor will mark these assignments. Each assignment has a weighting of 12.5%. The assignment component is worth 50% of the total course mark.

 

How to do your assignments

For each assignment, first read quickly through the description of the problem in the Assignment File. Make brief notes on what you believe are the key points raised. Next, carefully read the description two or three times while referring to your notes. Make sure that you have identified all the key points. Then, read the instructions that accompany the problem. These explain what you are required to do. Make sure you understand what is required and that your assignment provides what is required.

When you have completed the assignment, you should submit your assignment via the e-submission in the OLE. Make sure that each assignment is uploaded to the OLE before the due date. Marks may be deducted for work that is late without prior authorization. If, for any reason, you cannot complete your work on time, contact your tutor before the assignment is due. This is to discuss the possibility of an extension. Extensions will not be granted after the due date unless there are extremely exceptional circumstances.

You should use references other than your textbooks or work when researching the answers for your assignments. Make sure that you reference your work properly. If you do not, you commit plagiarism, and will be penalized severely. Plagiarism is the theft of somebody else's work or ideas. This applies just as much to using the work of other students as it does to the authors of books. If you use somebody else's ideas in your work, give them credit for it. You do this by referencing. In the body of the work, this appears as (Stallings, 2012) for example. At the end of your assignment, list all of your references alphabetically in a section called 'References'. Include the full name, title and date and place of publication. For instance, one way to cite a reference is:

Stallings, W (2012) Computer Organization and Architecture (9th Edition), Prentice Hall.

Tutors

There will be both face-to-face tutorials and on-line tutors support.

 

Tutorials

There will be regular tutorial sessions with your Tutor, supporting the teaching in the main texts. Exact information about the date, time and place of each tutorial will be provided in Stop Presses. Attendance at tutorials is optional.

 

Surgeries

Regular surgery sessions are scheduled throughout the course, a tutor will be available to answer specific problems that you may have with the course work. These 'surgery' sessions are intended to provide you with specific assistance with specific problems; the tutor will not run a tutorial, and will not have facilities for talking to more than a few students at a time. The surgeries provide the opportunity for you to resolve difficulties without having to wait for your next scheduled tutorial. Details of these surgery sessions will be sent to you in Stop Presses.

 

Online support

This course will use the Online Learning Environment (OLE) http://ole.hkmu.edu.hk. Please kindly read the OLE letter and OLE user guide to learn the detailed information which contains information relating to the course. The OLE will be a key resource for the students through the provision of live discussion and support features. The concept of a 'Support Zone' is used.

A note about the developers of this course

Dr Paul Kwok

Paul Kwok graduated from the University of Essex with a BSc degree with first class honours in Telecommunication Engineering in 1976 and obtained his PhD degree in Electrical Engineering from the University of Cambridge in 1979.

He joined the then-Monotype International in the Cambridge Science Park in 1979 and worked on the typesetting of Chinese text in a 1000 dpi laser phototypesetter. He has worked for the then-Hong Kong Polytechnic (Electronic Engineering), The Chinese University of Hong Kong (Computer Science), and the then-University of East Asia (Business Administration). Between 1986 and 1997, he was Associate Professor of Computer Science at the University of Calgary. From 1995-1999 he was Associate Professor and Course Director of Computer Engineering in the Department of Electrical and Electronic Engineering, The University of Hong Kong.

Dr Kwok is Professor in the School of Science & Technology, Hong Kong Metropolitan University.

 

Dr Wilson Hon-Wai Chu

Wilson Hon-Wai Chu received his BS (summa cum laude) in Electrical Engineering from Boston University in 1988. He also obtained MSE degrees in Electrical Engineering and Mathematical Science from the Johns Hopkins University in 1990 and 1992, respectively. He earned his PhD degree in Electronic Engineering from the Hong Kong University of Science and Technology in 1996. Dr. Chu was a CW Chu Foundation Scholar from 1985 to 1988.

Dr. Chu is an Assistant Professor in the Engineering Science programmes.

Dr. Chu is a Senior Member of IEEE and a Member of IEICE and Tau Beta Phi.