ethan

Time Operations in Go#

Operations related to time in Golang mainly involve the time package, with the core data structure being time.Time, as follows:

type Time struct {
    wall uint64
    ext  int64
    loc *Location
}

1. Functions Related to Time Retrieval#

1.1 Get Current Time#

// Returns the current time, note that the returned type is time.Time
now := time.Now()
fmt.Println(now)
// Current timestamp
fmt.Println(now.Unix())
// Nanosecond-level timestamp
fmt.Println(now.UnixNano())
// Decimal part of the timestamp in nanoseconds
fmt.Println(now.Nanosecond())

Output:

2021-01-10 14:56:15.930562 +0800 CST m=+0.000124449
1610261775
1610261775930562000
930562000

1.2 Return Current Year, Month, Day, Hour, Minute, Second, Day of the Week, and Day of the Year#

now := time.Now()
// Return date
year, month, day := now.Date()
fmt.Printf("year:%d, month:%d, day:%d\n", year, month, day)
// Year
fmt.Println(now.Year())
// Month
fmt.Println(now.Month())
// Day
fmt.Println(now.Day())
// Hour, Minute, Second
hour, minute, second := now.Clock()
fmt.Printf("hour:%d, minute:%d, second:%d\n", hour, minute, second)
// Hour
fmt.Println(now.Hour())
// Minute
fmt.Println(now.Minute())
// Second
fmt.Println(now.Second())
// Return day of the week
fmt.Println(now.Weekday())
// Return the corresponding day of the year
fmt.Println(now.YearDay())
// Return timezone
fmt.Println(now.Location())
// Return the day of the year
fmt.Println(now.YearDay())

1.3 Format Time#

Go provides a time formatting function Format(), and it is important to note that the Go time formatting template is not the common Y-m-d H:i:s, but rather 2006-01-02 15:04:05, which is also easy to remember (2006 1 2 3 4 5).

now := time.Now()
fmt.Println(now.Format("2006-01-02 15:04:05"))
fmt.Println(now.Format("2006-01-02"))
fmt.Println(now.Format("15:04:05"))
fmt.Println(now.Format("2006/01/02 15:04"))
fmt.Println(now.Format("15:04 2006/01/02"))

2. Convert Between Timestamps and Date Strings#

To convert a timestamp to a date format, you need to first convert the timestamp to time.Time type and then format it into a date format.

2.1 Return time.Time Type Based on Seconds and Nanoseconds#

now := time.Now()
layout := "2006-01-02 15:04:05"
t := time.Unix(now.Unix(), 0)    // Parameters are: seconds, nanoseconds
fmt.Println(t.Format(layout))

2.2 Return time.Time Type Based on Specified Time Using time.Date()#

now := time.Now()
layout := "2006-01-02 15:04:05"
// Return time.Time type based on specified time
// Specify year, month, day, hour, minute, second, nanosecond, timezone respectively
t := time.Date(2011, time.Month(3), 12, 15, 30, 20, 0, now.Location())
fmt.Println(t.Format(layout))

2.3 Parse Date String into time.Time Type#

t, _ := time.ParseInLocation("2006-01-02 15:04:05", time.Now().Format("2006-01-02 15:04:05"), time.Local)
fmt.Println(t)  
// Output 2021-01-10 17:28:50 +0800 CST
// time.Local specifies local time

When parsing, special attention should be paid to the issue of time zones:

fmt.Println(time.Now())
fmt.Println(time.Now().Location())
t, _ := time.Parse("2006-01-02 15:04:05", "2021-01-10 15:01:02")
fmt.Println(t)

Output:

2021-01-10 17:22:10.951904 +0800 CST m=+0.000094166
Local
2021-01-10 15:01:02 +0000 UTC

As can be seen, time.Now() uses CST (China Standard Time), while time.Parse() defaults to UTC (zero timezone), which is an 8-hour difference. Therefore, it is common to use time.ParseInLocation(), which allows specifying the time zone.

3. Calculate and Compare Dates#

When it comes to date calculations, one must mention a new type provided by the time package, Duration, which is defined in the source code as follows:

type Duration int64

The underlying type is int64, representing a time interval, with the unit being nanoseconds.

3.1 Time Calculations Within 24 Hours#

now := time.Now()
fmt.Println(now)
// 1 hour 1 minute 1 second later
t1, _ := time.ParseDuration("1h1m1s")
fmt.Println(t1)
m1 := now.Add(t1)
fmt.Println(m1)
// 1 hour 1 minute 1 second earlier
t2, _ := time.ParseDuration("-1h1m1s")
m2 := now.Add(t2)
fmt.Println(m2)
// 3 hours earlier
t3, _ := time.ParseDuration("-1h")
m3 := now.Add(t3 * 3)
fmt.Println(m3)
// 10 minutes later
t4, _ := time.ParseDuration("10m")
m4 := now.Add(t4)
fmt.Println(m4)
// Sub calculates the difference between two times
sub1 := now.Sub(m3)
fmt.Println(sub1.Hours())   // Difference in hours
fmt.Println(sub1.Minutes()) // Difference in minutes

Additionally, two functions time.Since() and time.Until() are introduced:

// Returns the time difference between the current time and t, the return value is Duration
time.Since(t Time) Duration
// Returns the time difference between t and the current time, the return value is Duration
time.Until(t Time) Duration

now := time.Now()
fmt.Println(now)
t1, _ := time.ParseDuration("-1h")
m1 := now.Add(t1)
fmt.Println(m1)
fmt.Println(time.Since(m1))
fmt.Println(time.Until(m1))

Output:

2021-01-10 20:41:48.668232 +0800 CST m=+0.000095594
2021-01-10 19:41:48.668232 +0800 CST m=-3599.999904406
1h0m0.000199007s
-1h0m0.000203035s

3.2 Time Calculations Beyond 24 Hours#

For time calculations beyond one day, time.AddDate() is used, with the function prototype:

func (t Time) AddDate(years int, months int, days int) Time

For example, to find out the time one year, one month, and one day later, you can do:

now := time.Now()
fmt.Println(now)
m1 := now.AddDate(1, 1, 1)
fmt.Println(m1)

For example, to get the time two days earlier:

now := time.Now()
fmt.Println(now)
m1 := now.AddDate(0, 0, -2)
fmt.Println(m1)

3.3 Date Comparison#

There are three types of date comparisons: before, after, and equal.

// If the time point represented by t is before u, return true; otherwise return false.
func (t Time) Before(u Time) bool
// If the time point represented by t is after u, return true; otherwise return false.
func (t Time) After(u Time) bool
// Compare whether the times are equal; return true if equal; otherwise return false.
func (t Time) Equal(u Time) bool

now := time.Now()
fmt.Println(now)
// 1 hour later
t1, _ := time.ParseDuration("1h")
m1 := now.Add(t1)
fmt.Println(m1)
fmt.Println(m1.After(now))
fmt.Println(now.Before(m1))
fmt.Println(now.Equal(m1))

Output:

2021-01-10 21:00:44.409785 +0800 CST m=+0.000186800
2021-01-10 22:00:44.409785 +0800 CST m=+3600.000186800
true
true
false

4. Common Examples#

Here are some common examples and function encapsulations.

4.1 Date Format to Timestamp#

func TimeStr2Time(fmtStr, valueStr, locStr string) int64 {
    loc := time.Local
    if locStr != "" {
        loc, _ = time.LoadLocation(locStr) // Set timezone
    }
    if fmtStr == "" {
        fmtStr = "2006-01-02 15:04:05"
    }
    t, _ := time.ParseInLocation(fmtStr, valueStr, loc)
    return t.Unix()
}

4.2 Get Current Time in Date Format#

func GetCurrentFormatStr(fmtStr string) string {
    if fmtStr == "" {
        fmtStr = "2006-01-02 15:04:05"
    }
    return time.Now().Format(fmtStr)
}

4.3 Timestamp to Date Format#

func Sec2TimeStr(sec int64, fmtStr string) string {
    if fmtStr == "" {
        fmtStr = "2006-01-02 15:04:05"
    }
    return time.Unix(sec, 0).Format(fmtStr)
}

Go-regexp Regular Expressions#

package main

import (
    "fmt"
    "regexp"
)

const text = "My email is [email protected]"

func main() {
    compile := regexp.MustCompile(`[a-zA-Z0-9]+@[a-zA-Z0-9]+\.[a-zA-Z0-9]+`)
    match := compile.FindString(text)
    fmt.Println(match)
}

Go Storage Basics — File IO Operations#

Two Major IO Categories#

The computing architecture includes CPU, memory, network, and IO. So what is IO? Generally understood as the abbreviation for Input and Output, simply put, it means input and output.

IO is divided into two types: network IO and storage IO (in fact, network IO and disk IO have fundamental differences in Go). Network IO corresponds to the process of network data transmission, which is the cornerstone of distributed systems, connecting discrete physical nodes through the network to form an organic system.

Storage IO corresponds to the process of storing data to physical media, usually referring to disks. Disks are generally divided into partitions, and a file system is formatted on top of them, so ordinary programmers most commonly see file IO.

In Golang, there are two ways to read and write files:

1. Standard library encapsulation: operating object File;
2. System calls: operating object fd;

Elements of Reading and Writing Data#

What is the core element of file reading and writing?

Simply put: reading a file means reading data from a specific location of the file on the disk into the memory buffer. Writing a file means writing data from the memory buffer to a specific location in the disk file.

Here, two keywords are noted:

1. Specific location;
2. Memory buffer;

How to understand specific location? How to specify the so-called specific location?

It's simple; you can identify a segment of location using [ offset, length ].

This is the IO offset and length, Offset and Length.

How to understand memory buffer?

Ultimately, who does the file data interact with directly? Memory. When writing, data is written from memory to the disk file, and when reading, data is read from the disk file to memory.

Essentially, the following IO functions revolve around these three elements: Offset, Length, and buffer.

Standard Library Encapsulation#

Reading and writing files in Go is very simple because Go has encapsulated a very convenient interface in the standard library os. The Go standard library's encapsulation of file IO is the recommended way to perform IO on files.

Open File (Open)#

func OpenFile(name string, flag int, perm FileMode) (*File, error)

After opening a file, you get a handle, which is the File structure, and subsequent read and write operations on the file are based on the File structure.

type File struct {
    *file // os specific
}

File reading and writing only requires operations on this handle structure.

Another hidden knowledge point must be mentioned: Offset. This is one of the three elements emphasized at the beginning of reading and writing. When opening (Open) a file, the current offset of the file is set to 0 by default, meaning that the starting position of IO is the very beginning of the file. For example, if at this point, 4K of data is written to the file, then it writes data at the position [0, 4K]. If there was already data there, it would overwrite it.

Unless the Open file specifies the O_APPEND option, the offset will be set to the end of the file, meaning that IO will start from the end of the file.

File Write Operations (Write)#

The file File handle object has two write methods:

The first: write a buffer to the file using the current file offset

func (f *File) Write(b []byte) (n int, err error)

Note: This write operation will cause the file offset to increase.

The second: write a buffer to the file from a specified file offset

func (f *File) WriteAt(b []byte, off int64) (n int, err error)

Note: This write operation will not update the file offset.

File Read Operations (Read)#

Corresponding to writing, the file File handle object has two read methods:

The first: read a buffer's data from the current file offset

func (f *File) Read(b []byte) (n int, err error)

Note: This read operation will cause the file offset to increase.

The second: read a buffer-sized data from a specified file offset

func (f *File) ReadAt(b []byte, off int64) (n int, err error)

Note: This read operation will not update the file offset.

Specify Offset (Seek)#

func (f *File) Seek(offset int64, whence int) (ret int64, err error)

This handle method allows users to specify the offset position of the file. This is easy to understand; for example, if the file starts at 0 bytes and 1M of data is written, the size becomes 1M, and the Offset moves back 1M, which is the default.

Now the Seek method allows the write offset to be positioned at any location, allowing data to be overwritten from anywhere.

Thus, in Go, file IO is very simple: first Open a file, get the File handle, and then you can use this handle to Write, Read, and Seek to perform IO.

Underlying Principles#

Go's standard library os provides extremely convenient encapsulation, and delving into the most primitive essence reveals the core: system calls.

Go's standard library file storage IO is based on system calls. You can follow the call of os.OpenFile:

The OpenFile function of the os library:

func OpenFile(name string, flag int, perm FileMode) (*File, error) {
    f, err := openFileNolog(name, flag, perm)
    if err != nil {
        return nil, err
    }
    f.appendMode = flag&O_APPEND != 0
    return f, nil
}

Take a look at the openFileNolog function:

func openFileNolog(name string, flag int, perm FileMode) (*File, error) {
    var r int
    for {
        var e error
        r, e = syscall.Open(name, flag|syscall.O_CLOEXEC, syscallMode(perm))
        if e == nil {
            break
        }
        if runtime.GOOS == "darwin" && e == syscall.EINTR {
            continue
        }
        return nil, &PathError{"open", name, e}
    }
    return newFile(uintptr(r), name, kindOpenFile), nil
}

You can see syscall.Open, this function gets an integer, which is the most common fd handle in C language, while the File structure is merely a layer of encapsulation based on this.

Consider why there is a standard library encapsulation layer?

Key Point: To shield the differences of operating systems, using this standard library means all operations are cross-platform. In other words, if there are features unique to special operating systems, then you won't find corresponding encapsulated IO operations in the os library.

So how to use system calls?

Directly use the syscall library, which is system calls. As the name suggests, system calls are strongly related to the operating system, as they are calling interfaces provided by the operating system, so system calls can lead to different characteristics and interfaces depending on the operating system.

Therefore, if you directly use the syscall library to make system calls, you need to bear the compatibility issues brought by the system yourself.

System Calls#

System calls have a basic encapsulation in syscall:

Open File#

func Open(path string, mode int, perm uint32) (fd int, err error) 

Read File#

func Read(fd int, p []byte) (n int, err error) func Pread(fd int, p []byte, offset int64) (n int, err error) 

File reading has two interfaces, one Read reads a buffer's data from the current default offset, and the Pread interface reads data from a specified position.

Consider this question: does Pread effectively equal Seek and Read combined?

No! The fundamental reason is that Seek + Read is two operations at the user level, while Pread, although it has the effect of Seek + Read, the semantics provided to the user is that Pread is an atomic operation. Another important difference is that Pread does not change the current file offset (ordinary Read calls will update the offset).

So, to summarize, **Pread** and sequentially calling **Seek** then calling **Read** have two important differences:

1. Pread provides the semantics of an atomic operation to the user; when calling Pread, the Seek and Read operations cannot be interrupted;
2. Pread calls do not update the current file offset;

Write File#

func Write(fd int, p []byte) (n int, err error) func Pwrite(fd int, p []byte, offset int64) (n int, err error) 

File writing also has two interfaces, Write and Pwrite, corresponding to Read and Pread. Similarly, Pwrite also has two differences:

1. Pwrite completes the semantics of Seek and Write as an atomic operation;
2. Pwrite calls do not update the current file offset;

Seek File#

func Seek(fd int, offset int64, whence int) (off int64, err error) 

This function call allows users to specify the offset (which will affect the read and write positions of Read and Write). Generally, each opened file has an associated "current file offset" (current file offset). The read (Read) and write (Write) operations start from the current file offset, and both Read and Write will cause the offset to increase, with the increase being the number of bytes read or written.

In summary: The core system calls for Open, Read, Write, and Seek show a clear difference from the standard IO library: the object of system call operations is an integer handle. The Open file gets an integer fd, and all subsequent IO operations are performed on this fd. This is obviously different from the standard library, where the os.OpenFile gets a File structure, and all IO operations are also performed on this structure.

Layered Architecture#

So what does the encapsulation layer generally look like? The Unix programming introduction has a diagram like this:

This diagram clearly illustrates the entire Unix architecture.

• The kernel is the core implementation, including functions for interacting with IO devices and hardware. The layer closely associated with the kernel is the system calls provided by the kernel, which provide a channel for user mode to kernel mode calls;

• For system calls, various languages' standard libraries have some encapsulation, such as C language's libc library, Go language's os, syscall libraries are similar in status, which is the so-called public library. The main purpose of this layer of encapsulation is to simplify the usage efficiency for ordinary programmers and shield system details, providing a basis for cross-platform (similarly, for the characteristics of cross-platform, many incompatible features may be omitted, which is why there is a need to directly call system calls);

• Of course, in the upper right corner, there is a gap; applications can use public function libraries, but they can also directly call system calls, but the complexity brought by this must be borne by the application itself. This demand is also common; the standard library encapsulates common things while also omitting many system call functions, in which case only system calls can be used to obtain them;

Summary#

1. IO is broadly divided into network IO and disk IO. For files, IO refers to read and write operations, where data is written from memory to disk and read from disk to memory;

2. The most commonly used file IO in Go is the os library. Using the Go encapsulated standard library, os.OpenFile opens, File.Write, File.Read performs read and write operations, and the operating object is the File structure;

3. The Go standard library's encapsulation of IO is to shield the complex system calls and provide a cross-platform usage posture. It also separately provides the syscall library, allowing programmers to decide whether to use richer system call functions, with the consequences borne by themselves;

4. The IO operation object of the Go standard library is File, while the IO operation object of system calls is fd (non-negative integer).

5. The default current offset of an Open file is 0 (the very beginning of the file). After adding the O_APPEND parameter, the offset will be at the end of the file. The Seek call can specify the file offset arbitrarily, thus affecting the position of file IO;

6. The Read and Write functions only have a buffer (the buffer has a length), and the offset uses the current file offset;

7. The system call effects of Pread and Pwrite are equivalent to Seek offset followed by Read and Write, but they have two important differences: the external semantics are atomic operations, and they do not update the current file offset;

Go - File Read and Write Operations#

Read and Write Files#

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

/*Clear existing content in the file and append*/
func main() {
    filePath := "D:\\fcofficework\\DNS\\1.txt"
    file, err := os.OpenFile(filePath, os.O_RDWR|os.O_APPEND, 0666)
    if err != nil {
        fmt.Printf("open file err = %v\n", err)
        return
    }
    /*Close file stream*/
    defer file.Close()
    /*Read*/
    reader := bufio.NewReader(file)
    for {
        str, err := reader.ReadString('\n')
        if err == io.EOF {
            break
        }
        fmt.Print(str)
    }
    /*Write to file*/
    str := "hello FCC您好！！！\r\n"
    writer := bufio.NewWriter(file)
    for i := 0; i < 5; i++ {
        writer.WriteString(str)
    }
    /*Since writer is buffered, it needs to be flushed to disk*/
    writer.Flush()
}

Copy File Content to New File#

package main

import (
    "fmt"
    "io/ioutil"
)

/*Copy contents of file1 to file2*/
func main() {
    file1Path := "D:\\fcofficework\\DNS\\1.txt"
    file2Path := "D:\\fcofficework\\DNS\\2.txt"
    data, err := ioutil.ReadFile(file1Path)
    if err != nil {
        fmt.Printf("read file err=%v", err)
        return
    }
    err = ioutil.WriteFile(file2Path, data, 0666)
    if err != nil {
        fmt.Printf("write file err=%v\n", err)
    }
}

Check if File or Directory Exists#

package main

import (
    "fmt"
    "os"
)

/*Check if file and directory exist*/
func PathExists(path string) (bool, error) {
    _, err := os.Stat(path)
    if err == nil {
        fmt.Println("Current file exists!")
        return true, nil
    }
    if os.IsNotExist(err) {
        fmt.Println("Current file does not exist!")
        return false, nil
    }
    return false, nil
}

func main() {
    path := "D:\\fcofficework\\2.txt"
    PathExists(path)
}

File Copy#

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

/*File copy*/

func CopyFile(dstFileName string, srcFileName string) (written int64, err error) {
    srcFile, err := os.Open(srcFileName)
    if err != nil {
        fmt.Printf("open file err=%v\n", err)
    }
    reader := bufio.NewReader(srcFile)

    dstFile, err := os.OpenFile(dstFileName, os.O_WRONLY|os.O_CREATE, 0666)
    if err != nil {
        fmt.Printf("open file err=%v\n", err)
        return
    }
    writer := bufio.NewWriter(dstFile)
    defer dstFile.Close()
    return io.Copy(writer, reader)
}

func main() {
    srcFile := "D:\\Photos\\Datapicture\\mmexport1530688562488.jpg"
    dstFile := "D:\\Photos\\1.jpg"
    _, err := CopyFile(dstFile, srcFile)
    if err == nil {
        fmt.Println("Copy completed!")
    } else {
        fmt.Println("Copy failed, err=", err)
    }
}

Read File and Count Characters#

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

/*Count characters in a file*/

type CharCount struct {
    /*Count of English characters*/
    ChCount int
    /*Count of digits*/
    NumCount int
    /*Count of spaces*/
    SpaceCount int
    /*Count of other characters*/
    OtherCount int
}

func main() {
    fileName := "D:\\fcofficework\\DNS\\1.txt"
    file, err := os.Open(fileName)
    if err != nil {
        fmt.Printf("open file err=%v\n", err)
        return
    }
    defer file.Close()
    var count CharCount
    reader := bufio.NewReader(file)
    for {
        str, err := reader.ReadString('\n')
        if err == io.EOF {
            break
        }
        for _, v := range str {
            switch {
            case v >= 'a' && v <= 'z':
                fallthrough
            case v >= 'A' && v <= 'Z':
                count.ChCount++
            case v == ' ' || v == '\t':
                count.SpaceCount++
            case v >= '0' && v <= '9':
                count.NumCount++
            default:
                count.OtherCount++
            }
        }
    }
    fmt.Printf("Character count: %v, Digit count: %v, Space count: %v, Other character count: %v",
        count.ChCount, count.NumCount, count.SpaceCount, count.OtherCount)
}

Three Ways to Read Files#

Read via os#

package main

import (
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("d:\\Photos\\Screenshots\\暗物质\\IMG_20180927_194619.jpg")
    if err != nil {
        fmt.Println("open file err", err)
    }
    fmt.Printf("file=%v", file)
    err1 := file.Close()
    if err1 != nil {
        fmt.Println("close file err = ", err1)
    }
}

Buffered Read File#

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

/*Buffered read file*/
func main() {
    file, err := os.Open("d:\\Photos\\Screenshots\\暗物质\\IMG_20180927_194619.jpg")
    if err != nil {
        fmt.Println("open file err", err)
    }
    defer file.Close()
    reader := bufio.NewReader(file)
    for {
        str, err := reader.ReadString('\n')
        if err == io.EOF {
            break
        }
        fmt.Print(str)
    }
    fmt.Println("File reading ended!")
}

Read via ioutil#

package main

import (
    "fmt"
    "io/ioutil"
)

func main() {
    file := "D:\\fcofficework\\DNS\\authorized_keys"
    content, err := ioutil.ReadFile(file)
    if err != nil {
        fmt.Printf("read file err=%v", err)
    }
    fmt.Printf("%v", string(content))
}

Writing to a File Case#

Write Content to a File, Create if Not Exists#

package main

import (
    "bufio"
    "fmt"
    "os"
)

/*Write content to a file, create if not exists*/
func main() {
    filePath := "D:\\fcofficework\\DNS\\1.txt"
    file, err := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE, 0666)
    if err != nil {
        fmt.Printf("open file err = %v\n", err)
        return
    }
    defer file.Close()
    str := "hello world\r\n"
    writer := bufio.NewWriter(file)
    for i := 0; i < 5; i++ {
        writer.WriteString(str)
    }
    /*Since writer is buffered, it needs to be flushed to disk*/
    writer.Flush()
}

Clear Existing Content in the File and Rewrite#

package main

import (
    "bufio"
    "fmt"
    "os"
)

/*Clear existing content in the file and rewrite*/
func main() {
    filePath := "D:\\fcofficework\\DNS\\1.txt"
    file, err := os.OpenFile(filePath, os.O_WRONLY|os.O_TRUNC, 0666)
    if err != nil {
        fmt.Printf("open file err = %v\n", err)
        return
    }
    defer file.Close()
    str := "hello FCC\r\n"
    writer := bufio.NewWriter(file)
    for i := 0; i < 5; i++ {
        writer.WriteString(str)
    }
    /*Since writer is buffered, it needs to be flushed to disk*/
    writer.Flush()
}

Clear Existing Content in the File and Append#

package main

import (
    "bufio"
    "fmt"
    "os"
)

/*Clear existing content in the file and append*/
func main() {
    filePath := "D:\\fcofficework\\DNS\\1.txt"
    file, err := os.OpenFile(filePath, os.O_WRONLY|os.O_APPEND, 0666)
    if err != nil {
        fmt.Printf("open file err = %v\n", err)
        return
    }
    defer file.Close()
    str := "hello FCC您好！！！\r\n"
    writer := bufio.NewWriter(file)
    for i := 0; i < 5; i++ {
        writer.WriteString(str)
    }
    /*Since writer is buffered, it needs to be flushed to disk*/
    writer.Flush()
}

Parse Command Line Arguments#

package main

import (
    "fmt"
    "os"
)

/*Parse command line arguments*/
func main() {
    fmt.Println("Command line arguments count:", len(os.Args))
    for i, v := range os.Args {
        fmt.Printf("args[%v]=%v\n", i, v)
    }
}
package main

import (
    "flag"
    "fmt"
)

/*Parse command line arguments*/
func main() {
    var user string
    var pwd string
    var host string
    var port int
    flag.StringVar(&user, "u", "", "Username, default is empty")
    flag.StringVar(&pwd, "pwd", "", "Password, default is empty")
    flag.StringVar(&host, "h", "localhost", "Hostname, default is empty")
    flag.IntVar(&port, "port", 3306, "Port number, default is empty")
    /*Convert*/
    flag.Parse()
    fmt.Printf("user=%v pwd=%v host=%v port=%v", user, pwd, host, port)
}

Go-json Serialization#

Serialization#

package main

import (
    "encoding/json"
    "fmt"
)

type Monster struct {
    Name     string  `json:"name"`
    Age      int     `json:"age"`
    Birthday string  `json:"birthday"`
    Sal      float64 `json:"sal"`
    Skill    string  `json:"skill"`
}

/*Struct serialization*/
func NewMinsterStruct() {
    monster := Monster{
        Name:     "Sun Wukong",
        Age:      500,
        Birthday: "2011-11-11",
        Sal:      8000.0,
        Skill:    "72 Transformations",
    }
    data, err := json.Marshal(&monster)
    if err != nil {
        fmt.Printf("Serialization error err：%v\n", err)
    }
    fmt.Printf("Map serialized result=%v\n", string(data))
}

/*Map serialization*/
func MapSerlizer() {
    var a map[string]interface{}
    a = make(map[string]interface{})
    a["name"] = "Bull Demon King"
    a["age"] = 10
    a["address"] = "Fire Cloud Cave"
    data, err := json.Marshal(a)
    if err != nil {
        fmt.Printf("Serialization error err：%v\n", err)
    }
    fmt.Printf("Monster serialized result=%v\n", string(data))
}

/*Slice serialization*/
func SliceSerlizer() {
    var slice []map[string]interface{}
    var m1 map[string]interface{}
    m1 = make(map[string]interface{})
    m1["name"] = "TGH"
    m1["age"] = "19"
    m1["address"] = "Beijing"
    slice = append(slice, m1)

    var m2 map[string]interface{}
    m2 = make(map[string]interface{})
    m2["name"] = "FCC"
    m2["age"] = "18"
    m2["address"] = [2]string{"Hua Fu", "Film Empire"}
    slice = append(slice, m2)

    data, err := json.Marshal(slice)
    if err != nil {
        fmt.Printf("Serialization error err：%v\n", err)
    }
    fmt.Printf("Slice serialized result=%v\n", string(data))
}

/*Basic data type serialization*/
func FloatSerlize() {
    var num1 float64 = 245.56
    data, err := json.Marshal(num1)
    if err != nil {
        fmt.Printf("Serialization error err：%v\n", err)
    }
    fmt.Printf("Basic data type serialization result=%v\n", string(data))
}

func main() {
    NewMinsterStruct()
    MapSerlizer()
    SliceSerlizer()
    FloatSerlize()
}
Map serialized result={"name":"Sun Wukong","age":500,"birthday":"2011-11-11","sal":8000,"skill":"72 Transformations"}
Monster serialized result={"address":"Fire Cloud Cave","age":10,"name":"Bull Demon King"}
Slice serialized result=[{"address":"Beijing","age":"19","name":"TGH"},{"address":["Hua Fu","Film Empire"],"age":"18","name":"FCC"}]
Basic data type serialization result=245.56

Deserialization#

package main

import (
    "encoding/json"
    "fmt"
)

type Monster struct {
    Name     string  `json:"name"`
    Age      int     `json:"age"`
    Birthday string  `json:"birthday"`
    Sal      float64 `json:"sal"`
    Skill    string  `json:"skill"`
}

func unmarshalStruct() {
    str := "{\"name\":\"Sun Wukong\",\"age\":500,\"birthday\":\"2011-11-11\",\"sal\":8000,\"skill\":\"72 Transformations\"}"
    var monster Monster
    err := json.Unmarshal([]byte(str), &monster)
    if err != nil {
        fmt.Printf("Deserialization failed err：%v\n", err)
    }
    fmt.Printf("Deserialized monster：%v\n", monster)
}

func unmarshallMap() {
    str := "{\"address\":\"Fire Cloud Cave\",\"age\":10,\"name\":\"Bull Demon King\"}"
    var a map[string]interface{}
    err := json.Unmarshal([]byte(str), &a)
    if err != nil {
        fmt.Printf("Deserialization failed err：%v\n", err)
    }
    fmt.Printf("Deserialized Map：%v\n", a)
}

func unmarshalSlice() {
    str := "[{\"address\":\"Beijing\",\"age\":\"19\",\"name\":\"TGH\"}," +
        "{\"address\":[\"Hua Fu\",\"Film Empire\"],\"age\":\"18\",\"name\":\"FCC\"}]"
    var slice []map[string]interface{}
    err := json.Unmarshal([]byte(str), &slice)
    if err != nil {
        fmt.Printf("Deserialization failed err：%v\n", err)
    }
    fmt.Printf("Deserialized Slice：%v\n", slice)
}

func main() {
    unmarshalStruct()
    unmarshallMap()
    unmarshalSlice()
}

Output:

Deserialized monster：{Sun Wukong 500 2011-11-11 8000 72 Transformations} 
Deserialized Map：map[address:Fire Cloud Cave age:10 name:Bull Demon King] 
Deserialized Slice：[map[address:Beijing age:19 name:TGH] map[address:[Hua Fu Film Empire] age:18 name:FCC]]

Go - Using the HTTP Package#

The web is a service based on the HTTP protocol, and the Go language provides a complete net/http package, which makes it very easy to set up a runnable web service. At the same time, this package allows simple settings and operations for web routing, static files, templates, cookies, and other data.

Establishing a Web Server with the HTTP Package#

package main
import (
    "fmt"
    "net/http"
    "strings"
    "log"
)
func sayhelloName(w http.ResponseWriter, r *http.Request) {
    r.ParseForm()  // Parse parameters, by default it will not parse
    fmt.Println(r.Form)  // This information is output to the server's print information
    fmt.Println("path", r.URL.Path)
    fmt.Println("scheme", r.URL.Scheme)
    fmt.Println(r.Form["url_long"])
    for k, v := range r.Form {
        fmt.Println("key:", k)
        fmt.Println("val:", strings.Join(v, ""))
    }
    fmt.Fprintf(w, "Hello golang!") // What is written to w is output to the client
}
func main() {
    http.HandleFunc("/", sayhelloName) // Set the access route
    err := http.ListenAndServe(":8080", nil) // Set the listening port
    if err != nil {
        log.Fatal("ListenAndServe: ", err)
    }
}

After building the above code and executing web.exe, it is already listening for HTTP link requests on port 8080.

Entering http://localhost:8080 in the browser

You can see the browser page outputs Hello golang!

Entering the address in the browser:

http://localhost:8080/?url_long=var1&url_long=var2

You can see what the browser outputs.

From the above code, it is very simple to write a web server; you only need to call two functions from the http package.

Requesting a Page Using the HTTP Package#

package main

import (
    "fmt"
    "net/http"
    "net/http/httputil"
)

func main() {
    request, err := http.NewRequest(http.MethodGet, "http://www.imooc.com", nil)
    if err != nil {
        panic(err)
    }
    request.Header.Add("User-Agent",
        "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1")

    client := http.Client{
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            fmt.Println("Redirect:", req)
            return nil
        },
    }

    resp, err := client.Do(request)
    //resp, err := http.DefaultClient.Do(request)
    //resp, err := http.Get("http://www.imooc.com")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    s, err := httputil.DumpResponse(resp, true)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(s))
}

The program runs and prints out the HTML content.

Pitfalls of the net/http Package — I/O Timeout#

Problem#

Let's look at a piece of everyday code.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net"
    "net/http"
    "time"
)

var tr *http.Transport

func init() {
    tr = &http.Transport{
        MaxIdleConns: 100,
        Dial: func(netw, addr string) (net.Conn, error) {
            conn, err := net.DialTimeout(netw, addr, time.Second*2) // Set connection timeout
            if err != nil {
                return nil, err
            }
            err = conn.SetDeadline(time.Now().Add(time.Second * 3)) // Set send and receive data timeout
            if err != nil {
                return nil, err
            }
            return conn, nil
        },
    }
}

func main() {
    for {
        _, err := Get("http://www.baidu.com/")
        if err != nil {
            fmt.Println(err)
            break
        }
    }
}


func Get(url string) ([]byte, error) {
    m := make(map[string]interface{})
    data, err := json.Marshal(m)
    if err != nil {
        return nil, err
    }
    body := bytes.NewReader(data)
    req, _ := http.NewRequest("Get", url, body)
    req.Header.Add("content-type", "application/json")

    client := &http.Client{
        Transport: tr,
    }
    res, err := client.Do(req)
    if res != nil {
        defer res.Body.Close()
    }
    if err != nil {
        return nil, err
    }
    resBody, err := ioutil.ReadAll(res.Body)
    if err != nil {
        return nil, err
    }
    return resBody, nil
}

What this does is relatively simple: it loops to request http://www.baidu.com/ and waits for a response.

It seems there is no problem.

The code runs and can indeed send and receive messages normally.

However, this piece of code runs for a while and will encounter an i/o timeout error.

This is actually a problem that was recently investigated, and it is found that this pitfall is relatively easy to step on, and the code is simplified here.

In actual production, when the Golang service initiates an HTTP call, although the http.Transport sets a 3s timeout, it occasionally reports an i/o timeout error.

However, when checking the downstream service, it is found that the downstream service has actually returned in 100ms.

Troubleshooting#

Analysis of the message body changes corresponding to the five-layer network protocol

It is very strange that the server shows that the processing time is only 100ms, and the client timeout is set to 3s, so how can an i/o timeout error occur?

Here are two possible speculations.

• The logs printed by the server are actually just the logs printed by the server application layer. However, after the client application layer sends data, it still goes through the client's transport layer, network layer, data link layer, and physical layer, and then through the server's physical layer, data link layer, network layer, transport layer to the server's application layer. The processing time on the server takes 100ms, and the remaining time of 3s-100ms may be spent on various layers in the entire process. For example, in the case of poor network conditions, the transport layer TCP may drop packets and retransmit them.
• The network is fine, and the link from the client to the server takes about 100ms. The client processing logic causes the timeout.

In general, when encountering problems, most of the time it is not a lower-level network issue. It is better to boldly suspect that it is your own problem. If you are not convinced, you can capture packets to check.

Capture packet results

Analyzing from the beginning of the three-way handshake (the place marked in red).

To the last occurrence of the timeout error i/o timeout (the place marked in blue).

From the time column, from 7 to 10, it indeed takes 3s. Moreover, looking at the lower right corner of the blue box, it is a Reset connection from port 51169 to port 80.

Port 80 is the server's port. In other words, the client actively disconnects the connection after 3s of timeout.

However, looking closely at the first three-way handshake to the last client timeout active disconnection, there are actually many HTTP requests in between.

Going back to check the way the timeout is set in the code.

tr = &http.Transport{
    MaxIdleConns: 100,
    Dial: func(netw, addr string) (net.Conn, error) {
        conn, err := net.DialTimeout(netw, addr, time.Second*2) // Set connection timeout
        if err != nil {
            return nil, err
        }
        err = conn.SetDeadline(time.Now().Add(time.Second * 3)) // Set send and receive data timeout
        if err != nil {
            return nil, err
        }
        return conn, nil
    },
}

Here, the 3s timeout is actually counted after the connection is established, not from the start of the single call.

Looking at the comments, it says:

SetDeadline sets the read and write deadlines associated with the connection.

Timeout Reason#

As we know, HTTP is an application layer protocol, and the transport layer uses the TCP protocol.

Before HTTP 1.0, short connections were used by default, and each request would establish a TCP connection, send and receive data, and then disconnect.

Each TCP connection requires a three-way handshake. Each disconnection requires a four-way handshake.

In fact, there is no need to establish a new connection every time; it is better to keep the established connection open and reuse it for each data transmission.

Thus, HTTP 1.1 defaults to using long connections. Relevant information can be found in the previous article.

So the Go standard library is also compatible with this implementation.

By establishing a connection pool, a TCP long connection is established for each domain name, for example, http://baidu.com and http://golang.com are two different domain names.

The first time you access http://baidu.com, a connection will be established, and after use, it will be placed in the idle connection pool. The next time you want to access http://baidu.com, this connection will be taken out from the pool for reuse.

Reuse long connections

Why emphasize the same domain: a domain name will establish a connection, and a connection corresponds to a read goroutine and a write goroutine. It is precisely because it is the same domain that 3 goroutines will leak; if it is different domains, it will leak 1+2*N goroutines, where N is the number of domain names.

Assuming the first request takes 100ms, after each request to http://baidu.com, the connection is placed back into the connection pool for reuse, repeating 29 times, taking 2900ms.

When the 30th request occurs, the connection has already taken 3000ms from establishment to server return, just hitting the 3s timeout threshold set, at which point the client will report a timeout error.

Even if the server processes quickly, the client sets a long timeout, there will always be a moment when the program will report a timeout error.

Correct Approach#

Originally, the expectation was to set a timeout for each call, rather than for the entire connection.

Additionally, the reason for the problem mentioned above is that a timeout was set for the long connection, and the long connection would be reused.

Based on these two points, the code can be modified.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "time"
)

var tr *http.Transport

func init() {
    tr = &http.Transport{
        MaxIdleConns: 100,
        // The code below is removed
        //Dial: func(netw, addr string) (net.Conn, error) {
        //  conn, err := net.DialTimeout(netw, addr, time.Second*2) // Set connection timeout
        //  if err != nil {
        //      return nil, err
        //  }
        //  err = conn.SetDeadline(time.Now().Add(time.Second * 3)) // Set send and receive data timeout
        //  if err != nil {
        //      return nil, err
        //  }
        //  return conn, nil
        //},
    }
}


func Get(url string) ([]byte, error) {
    m := make(map[string]interface{})
    data, err := json.Marshal(m)
    if err != nil {
        return nil, err
    }
    body := bytes.NewReader(data)
    req, _ := http.NewRequest("Get", url, body)
    req.Header.Add("content-type", "application/json")

    client := &http.Client{
        Transport: tr,
        Timeout: 3*time.Second,  // Timeout is set here, which is the timeout for each call
    }
    res, err := client.Do(req) 
    if res != nil {
        defer res.Body.Close()
    }
    if err != nil {
        return nil, err
    }
    resBody, err := ioutil.ReadAll(res.Body)
    if err != nil {
        return nil, err
    }
    return resBody, nil
}

func main() {
    for {
        _, err := Get("http://www.baidu.com/")
        if err != nil {
            fmt.Println(err)
            break
        }
    }
}

Looking at the comments, there are two points of modification:

• The timeout settings for establishing connections in http.Transport are removed.
• When initiating an HTTP request, the http.Client is created, and at this time, the timeout setting is added. This timeout can be understood as the timeout for a single request.

At this point, the code is fixed, and the problem in actual production is resolved.

In the example code, if you run it, you may still encounter the following error:

Get http://www.baidu.com/: EOF

This is because the calls are too aggressive, and http://www.baidu.com actively closes the connection, which can be understood as a rate-limiting measure to protect the server. After all, if everyone does this, the server will crash...

The solution is simple: just add a sleep interval between each HTTP call.

At this point, the problem is resolved, and the following will analyze the reasons for the problem at the source code level.

Source Code Analysis#

Using Go version 1.12.7.

Starting from initiating a network request, let's follow the call.

res, err := client.Do(req)
func (c *Client) Do(req *Request) (*Response, error) {
    return c.do(req)
}

func (c *Client) do(req *Request) {
    // ...
    if resp, didTimeout, err = c.send(req, deadline); err != nil {
        // ...
    }
    // ...  
}  
func send(ireq *Request, rt RoundTripper, deadline time.Time) {
    // ...    
    resp, err = rt.RoundTrip(req)
    // ...  
}

// Focus on this function, returning a long connection
/src/net/http/roundtrip.go: 16
func (t *Transport) RoundTrip(req *Request) (*Response, error) {
    return t.roundTrip(req)
}

func (t *Transport) roundTrip(req *Request) (*Response, error) {
    // Try to get an idle connection for the HTTP connection
    pconn, err := t.getConn(treq, cm)
    // ...
}

// Pay attention to the following two points
func (t *Transport) getConn(treq *transportRequest, cm connectMethod) (*persistConn, error) {
    // Omitted a lot of logic, only focus on the following two points
    // If there is an idle connection, return it
    pc := <-t.getIdleConnCh(cm)

    // If not, create a connection
    pc, err := t.dialConn(ctx, cm)

}

When the first HTTP request is initiated, there will definitely be no idle connection, and a new connection will be established. At the same time, a read goroutine and a write goroutine will be created.

Read and write goroutines

Note that in the code above, t.dial(ctx, "tcp", cm.addr()) will execute the following code:

func (t *Transport) dial(ctx context.Context, network, addr string) (net.Conn, error) {
    // ...
    c, err := t.Dial(network, addr)
    // ...
}

The timeout setting here will be executed in the following code:

func (c *conn) SetDeadline(t time.Time) error {
    //...
    n, err := c.fd.Read(b)
    // ...
}

func (fd *netFD) Read(p []byte) (n int, error) {
    n, err = fd.pfd.Read(p)
    // ...
}

func (pd *pollDesc) waitRead(isFile bool) error {
    return pd.wait('r', isFile)
}

This will set a timer event when the timeout occurs, and the timer event will be registered.

Read and write goroutine timer event

When the first request is called, a connection is established, and a timeout event is registered. Assuming the time is set to 3s, this event will occur after 3s, executing the registered function. The registered function is netpollDeadline. Note that this netpollDeadline will set pd.rd = -1, which indicates that the read deadline of the network polling file descriptor is set.

At this point, the code will receive the i/o timeout error.

Summary#

1. Do not set timeouts in http.Transport, as that is the connection timeout, not the request timeout. Otherwise, it may lead to inexplicable i/o timeout errors.
2. The request timeout should be set when creating the client.

Unicode Related Packages in Go#

Unicode Related Packages in Go#

Go language simplifies complex encoding issues, greatly reducing the mental burden on programmers. To facilitate the processing of Unicode strings, the Go standard library provides three packages: unicode, unicode/utf8, and unicode/utf16.

Here is a brief introduction to the functions of these three packages:

• unicode: Provides data and functions to test certain properties of Unicode code points (stored as runes).

• unicode/utf8: Used to handle UTF-8 encoded text, providing some constants and functions, including conversions between runes (code points) and UTF-8 byte sequences.

• unicode/utf16: Has fewer functions, mainly for encoding and decoding UTF-16 sequences.

The string representation in Go.

In Go, string literals have four forms, for example, "徐新华" can be written as:

s1 := "徐新华"
s2 := "\u5F90\u65B0\u534E"
s3 := "\U00005F90\U000065B0\U0000534E"
s4 := "\xe5\xbe\x90\xe6\x96\xb0\xe5\x8d\x8e"

Simply put, \u is followed by four hexadecimal numbers, \U is followed by eight hexadecimal numbers. The \u or \U represents that what follows is a Unicode code point. The \x is followed by two hexadecimal numbers, and these hexadecimal numbers are not Unicode code points but UTF-8 encoded.

The following code is helpful for understanding:

package main
import (
    "fmt"
    "unicode/utf8"
)
func main() {
    s := `徐新华`
    var (
        buf = make([]byte, 4)
        n   int
    )
    fmt.Println("Character\tUnicode Code Point\tUTF-8 Hex Encoding\tUTF-8 Binary Encoding")
    for _, r := range s {
        n = utf8.EncodeRune(buf, r)
        fmt.Printf("%q\t%U\t\t%X\t\t%b\n", r, r, buf[:n], buf[:n])
    }

    s2 := "\u5F90\u65B0\u534E"
    s3 := "\U00005F90\U000065B0\U0000534E"
    s4 := "\xe5\xbe\x90\xe6\x96\xb0\xe5\x8d\x8e"

    fmt.Println(s2)
    fmt.Println(s3)
    fmt.Println(s4)
}

Running result:

Character Unicode Code Point UTF-8 Hex Encoding UTF-8 Binary Encoding
'徐' U+5F90  E5BE90  [11100101 10111110 10010000]
'新' U+65B0  E696B0  [11100110 10010110 10110000]
'华' U+534E  E58D8E  [11100101 10001101 10001110]
徐新华
徐新华
徐新华

Additionally, regarding other aspects of string processing, such as encoding conversion, you can find it at https://pkg.go.dev/golang.org/x/text.

Endianness (Little Endian and Big Endian)#

When a character is stored using multiple bytes, it involves which comes first and which comes last. Taking the Chinese character "徐" as an example, the Unicode code point is 5F90, which needs to be stored using two bytes, one byte is 5F, and the other byte is 90. In storage, 5F comes first, and 90 comes last, which is the Big Endian way; 90 comes first, and 5F comes last, which is the Little Endian way.

These two strange names come from the British writer Swift's "Gulliver's Travels." In the book, a civil war broke out in the land of little people, and the war was caused by the debate over whether to break an egg from the big end (Big-endian) or the small end (Little-endian). This led to six wars, one emperor lost his life, and another emperor lost his throne.

The first byte in front is "big-endian," while the second byte in front is "little-endian."

Naturally, a question arises: how does a computer know which encoding method a file uses?

The Unicode specification defines that each file's front must include a character that indicates the encoding order, called the "zero-width non-joiner" (zero width no-break space), represented by FEFF. This is exactly two bytes, and FF is one greater than FE.

If the first two bytes of a text file are FE FF, it indicates that the file uses big-endian; if the first two bytes are FF FE, it indicates that the file uses little-endian.

However, from the explanation of UTF-8 encoding above, it can be seen that although UTF-8 has multiple bytes to represent a character, the order is fixed, and there is no byte order issue. Under Unix systems, UTF-8 has no leading characters, but in Windows, the Notepad saves UTF-8 files with BOM (Byte Order Mark), which is EF BB BF, these three bytes. Regarding this, Rob Pike, the father of Unicode, clearly stated that UTF-8 does not need BOM, so initially, Go source files were not allowed to have BOM; otherwise, they would not compile, but now it is allowed. However, it is still recommended not to carry BOM.

UTF-8 with BOM is not for distinguishing byte order but to more conveniently identify that this is a UTF-8 file.

Go Standard Library - Unsafe Package#

1. Unsafe Package#

1.1. ArbitraryType#

The unsafe package defines an ArbitraryType, representing any Go expression.

type ArbitraryType int

1.2. Pointer#

Definition of Pointer:

type Pointer *ArbitraryType

Pointer represents a pointer to any type, and there are four operations that only apply to Pointer but not to other types.

• A pointer value of any type can be converted to a Pointer.

• A Pointer can be converted to a pointer value of any type.

• A uintptr can be converted to a Pointer.

• A Pointer can also be converted to a uintptr.

Thus, Pointer can bypass the type system and directly point to any type. Therefore, it must be used with great caution.

Regarding the rules for using Pointer, code that does not use these rules is unusable or will be unusable in the future.

1.2.1. Use Pointer as an intermediary to convert *T1 to *T2#

The premise is that the size of T2 does not exceed T1, and both have the same memory distribution.

func Float64bits(f float64) uint64 {  return *(*uint64)(unsafe.Pointer(&f))}

1.2.2. Convert Pointer to uintptr#

Converting Pointer to uintptr will produce an int variable pointing to the type value. It is commonly used to print a uintptr.

Converting uintptr back to Pointer is unusable.

This is because uintptr is an integer value, not a reference. In other words, uintptr and pointers have no relationship. It can be said that it returns the value of the address pointed to by Pointer to uintptr, even if the value in uintptr corresponds to the address's object is updated or deleted, uintptr will not change.

1.2.3. Convert Pointer to uintptr, then convert back to Pointer, with uintptr numerical operations#

If Pointer points to an allocated object, then the following conversion can move the Pointer forward.

 p = unsafe.Pointer(uintptr(p) + offset)

The most common use is to point to different fields in a structure or elements in an array.

// equivalent to f := unsafe.Pointer(&s.f)
 f := unsafe.Pointer(uintptr(unsafe.Pointer(&s)) + unsafe.Offsetof(s.f))
// equivalent to e := unsafe.Pointer(&x[i])
 e := unsafe.Pointer(uintptr(unsafe.Pointer(&x[0])) + i*unsafe.Sizeof(x[0]))

This can be used to move the pointer forward or backward by adding or subtracting offset. After moving the pointer, it should still point to the memory range.

Moving Pointer beyond its original memory allocation range is unusable, such as:

// INVALID: end points outside allocated space.
 var s thing
 end = unsafe.Pointer(uintptr(unsafe.Pointer(&s)) + unsafe.Sizeof(s))

 // INVALID: end points outside allocated space.
 b := make([]byte, n)
 end = unsafe.Pointer(uintptr(unsafe.Pointer(&b[0])) + uintptr(n))

Of course, the following code is also incorrect because uintptr cannot be stored in a variable:

// INVALID: uintptr cannot be stored in variable
// before conversion back to Pointer.
u := uintptr(p)
p = unsafe.Pointer(u + offset)

Pointer must point to an already allocated object and cannot be nil:

// INVALID: conversion of nil pointer
u := unsafe.Pointer(nil)
p := unsafe.Pointer(uintptr(u) + offset)

1.2.4. When calling syscall.Syscall, Pointer must be converted to uintptr#

The Syscall function in the syscall package passes uintptr parameters to the operating system, and then based on the relevant information of the call, the corresponding uintptr is converted back to a pointer.

If a pointer parameter must be converted to uintptr as a parameter, this conversion can only be completed in the parameter expression within the function call, because uintptr cannot be stored in a variable.

syscall.Syscall(SYS_READ, uintptr(fd), uintptr(unsafe.Pointer(p)), uintptr(n))

When the compiler processes pointers in function calls, the object pointed to by that pointer will be retained until the function call ends, even if that object is not used during the function call.

The following is incorrect code because the uintptr cannot be stored in a variable before implicit conversion back to Pointer during the system call.

// INVALID: uintptr cannot be stored in variable
// before implicit conversion back to Pointer during system call.
u := uintptr(unsafe.Pointer(p))
syscall.Syscall(SYS_READ, uintptr(fd), u, uintptr(n))

1.2.5. Convert the result of reflect.Value.Pointer or reflect.Value.UnsafeAddr from uintptr to Pointer#

The Pointer method and UnsafeAddr method of the Value in the reflect package return uintptr instead of Pointer, allowing the caller to convert to any type without importing the unsafe package. This means that the return value of these two methods must be converted to Pointer to be usable:

p := (*int)(unsafe.Pointer(reflect.ValueOf(new(int)).Pointer()))

Since the return values of these two function calls are uintptr, they also cannot be stored in variables.

1.2.6. The Data field of reflect.SliceHeader or reflect.StringHeader can be converted to and from Pointer#

As mentioned earlier, the return of uintptr is for the caller to directly perform different types of conversions without importing the unsafe package. This means that SliceHeader and StringHeader can only be used when Data is used as *SliceHeader and *StringHeader, and cannot be used in their struct form.

var s string
hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
hdr.Data = uintptr(unsafe.Pointer(p))              // case 6 (this case)
hdr.Len = n

In general, SliceHeader and StringHeader can only be used as *SliceHeader and *StringHeader, and cannot be used in their struct form.

 // INVALID: a directly-declared header will not hold Data as a reference.
 var hdr reflect.StringHeader
 hdr.Data = uintptr(unsafe.Pointer(p))
 hdr.Len = n
 s := *(*string)(unsafe.Pointer(&hdr)) // p possibly already lost

1.3 Sizeof Function#

Definition:

func Sizeof(x ArbitraryType) uintptr

Directly copying the content from the standard documentation, the same applies below.

Sizeof returns the number of bytes occupied by the data of type v itself. The return value is the number of bytes occupied by the "top-level" data. For example, if v is a slice, it will return the size of the slice descriptor, not the size of the memory referenced by the underlying slice.

1.4 Alignof#

Definition:

func Alignof(v ArbitraryType) uintptr

Alignof returns the alignment of type v (i.e., the number of bytes occupied by type v in memory); if it is in the form of a field of a struct type, it will return the alignment of field f in that struct.

1.5 Offsetof#

Definition:

func Offsetof(v ArbitraryType) uintptr

Offsetof returns the offset of the field represented by type v in the struct, and it must be in the form of a field of a struct type. In other words, it returns the number of bytes between the start of the struct and the start of that field.

Summary#

The difference between Pointer and uintptr in 1.2:

Assuming there is a variable in memory a := 1

Then in p := Pointer(&a), p contains the actual address of a, assuming it is 1000, and when a moves in memory, p's address value will also update in real-time.

While uintptr(p) is just 1000, which is the address of a, but when a moves in memory, the value obtained from uintptr will not change, it will always be 1000.

This is also why the uintptr passed to syscall.Syscall must represent a pointer to an object, and that object must be retained in memory and cannot move; otherwise, uintptr will no longer point to the original object, leading to memory leaks.

Another point is that uintptr cannot be stored in a variable; it must be converted using Pointer before it can be stored.

Encoding Analysis#

Background#

The HTTP protocol is based on text transmission, character encoding transforms text into binary, and binary encoding transforms binary back into text. The TCP protocol is based on binary transmission, and data reading needs to handle byte order. This article will introduce common character encoding, binary encoding, and byte order, and explore their implementation in Golang.

Character Encoding#

Introduction: How to turn "Hello world" into bytes?

• Step 1: Obtain the complete character set to be represented (character table).

• Step 2: Assign an integer number to each character (encoding character set).

• Step 3: Map the numbers to fixed-length bit values (character encoding table).

Characters refer to various letters and symbols, including the characters of various national languages, punctuation marks, graphic symbols, numbers, etc. There are a total of 5651 languages used worldwide, among which there are 13 languages with more than 50 million speakers, and each language has its own characters. In Chinese, a Chinese character is a character. In English, a letter is a character. Even invisible characters can be characters (such as control characters). The collection of characters is the character table, such as the English alphabet, the Arabic numeral table. The ASCII code table contains a total of 128 characters.

Encoding Character Set (CCS: Coded Character Set)#

Assign a number (code point) to each character in the character table, which results in the encoding character set. Common character sets include ASCII character set, Unicode character set, GB2312 character set, BIG5 character set, GB18030 character set, etc. The ASCII character set contains a total of 128 characters, including 94 printable characters (52 uppercase and lowercase English letters, 10 Arabic numerals, and 32 Western symbols) and 34 control or communication-specific characters, with code point values ranging from [0, 128), as shown in the figure below.

![img](ip