Bump deps

```
* use go 1.17 semantics in go.mod
* pin tonistiigi/fsutil dep to match docker/docker's dep
go get -u ./...
go mod vendor
```
This commit is contained in:
Jason Hall
2022-01-21 16:58:24 +00:00
committed by GitHub
parent a59970f4ac
commit c19ffc26a2
3595 changed files with 429699 additions and 120494 deletions
+2
View File
@@ -0,0 +1,2 @@
sftpfs/file1
sftpfs/test/
+26 -21
View File
@@ -1,21 +1,26 @@
sudo: false
language: go
go:
- 1.9
- "1.10"
- tip
os:
- linux
- osx
matrix:
allow_failures:
- go: tip
fast_finish: true
script:
- go build
- go test -race -v ./...
sudo: false
language: go
arch:
- amd64
- ppc64e
go:
- "1.14"
- "1.15"
- "1.16"
- tip
os:
- linux
- osx
matrix:
allow_failures:
- go: tip
fast_finish: true
script:
- go build -v ./...
- go test -count=1 -cover -race -v ./...
- go vet ./...
- FILES=$(gofmt -s -l . zipfs sftpfs mem tarfs); if [[ -n "${FILES}" ]]; then echo "You have go format errors; gofmt your changes"; exit 1; fi
+24 -34
View File
@@ -6,7 +6,7 @@ A FileSystem Abstraction System for Go
# Overview
Afero is an filesystem framework providing a simple, uniform and universal API
Afero is a filesystem framework providing a simple, uniform and universal API
interacting with any filesystem, as an abstraction layer providing interfaces,
types and methods. Afero has an exceptionally clean interface and simple design
without needless constructors or initialization methods.
@@ -18,7 +18,7 @@ and benefit of the os and ioutil packages.
Afero provides significant improvements over using the os package alone, most
notably the ability to create mock and testing filesystems without relying on the disk.
It is suitable for use in a any situation where you would consider using the OS
It is suitable for use in any situation where you would consider using the OS
package as it provides an additional abstraction that makes it easy to use a
memory backed file system during testing. It also adds support for the http
filesystem for full interoperability.
@@ -33,7 +33,7 @@ filesystem for full interoperability.
* Support for compositional (union) file systems by combining multiple file systems acting as one
* Specialized backends which modify existing filesystems (Read Only, Regexp filtered)
* A set of utility functions ported from io, ioutil & hugo to be afero aware
* Wrapper for go 1.16 filesystem abstraction `io/fs.FS`
# Using Afero
@@ -41,8 +41,8 @@ Afero is easy to use and easier to adopt.
A few different ways you could use Afero:
* Use the interfaces alone to define you own file system.
* Wrap for the OS packages.
* Use the interfaces alone to define your own file system.
* Wrapper for the OS packages.
* Define different filesystems for different parts of your application.
* Use Afero for mock filesystems while testing
@@ -79,11 +79,11 @@ would.
So if my application before had:
```go
os.Open('/tmp/foo')
os.Open("/tmp/foo")
```
We would replace it with:
```go
AppFs.Open('/tmp/foo')
AppFs.Open("/tmp/foo")
```
`AppFs` being the variable we defined above.
@@ -94,6 +94,7 @@ AppFs.Open('/tmp/foo')
File System Methods Available:
```go
Chmod(name string, mode os.FileMode) : error
Chown(name string, uid, gid int) : error
Chtimes(name string, atime time.Time, mtime time.Time) : error
Create(name string) : File, error
Mkdir(name string, perm os.FileMode) : error
@@ -227,7 +228,7 @@ operation and a mock filesystem during testing or as needed.
```go
appfs := afero.NewOsFs()
appfs.MkdirAll("src/a", 0755))
appfs.MkdirAll("src/a", 0755)
```
## Memory Backed Storage
@@ -241,7 +242,7 @@ safely.
```go
mm := afero.NewMemMapFs()
mm.MkdirAll("src/a", 0755))
mm.MkdirAll("src/a", 0755)
```
#### InMemoryFile
@@ -258,6 +259,18 @@ system using InMemoryFile.
Afero has experimental support for secure file transfer protocol (sftp). Which can
be used to perform file operations over a encrypted channel.
### GCSFs
Afero has experimental support for Google Cloud Storage (GCS). You can either set the
`GOOGLE_APPLICATION_CREDENTIALS_JSON` env variable to your JSON credentials or use `opts` in
`NewGcsFS` to configure access to your GCS bucket.
Some known limitations of the existing implementation:
* No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
* No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
* Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.
## Filtering Backends
### BasePathFs
@@ -306,7 +319,7 @@ Any Afero FileSystem can be used as an httpFs.
```go
httpFs := afero.NewHttpFs(<ExistingFS>)
fileserver := http.FileServer(httpFs.Dir(<PATH>)))
fileserver := http.FileServer(httpFs.Dir(<PATH>))
http.Handle("/", fileserver)
```
@@ -380,8 +393,6 @@ The following is a short list of possible backends we hope someone will
implement:
* SSH
* ZIP
* TAR
* S3
# About the project
@@ -406,28 +417,7 @@ Googles very well.
## Release Notes
* **0.10.0** 2015.12.10
* Full compatibility with Windows
* Introduction of afero utilities
* Test suite rewritten to work cross platform
* Normalize paths for MemMapFs
* Adding Sync to the file interface
* **Breaking Change** Walk and ReadDir have changed parameter order
* Moving types used by MemMapFs to a subpackage
* General bugfixes and improvements
* **0.9.0** 2015.11.05
* New Walk function similar to filepath.Walk
* MemMapFs.OpenFile handles O_CREATE, O_APPEND, O_TRUNC
* MemMapFs.Remove now really deletes the file
* InMemoryFile.Readdir and Readdirnames work correctly
* InMemoryFile functions lock it for concurrent access
* Test suite improvements
* **0.8.0** 2014.10.28
* First public version
* Interfaces feel ready for people to build using
* Interfaces satisfy all known uses
* MemMapFs passes the majority of the OS test suite
* OsFs passes the majority of the OS test suite
See the [Releases Page](https://github.com/spf13/afero/releases).
## Contributing
+4 -1
View File
@@ -91,9 +91,12 @@ type Fs interface {
// The name of this FileSystem
Name() string
//Chmod changes the mode of the named file to mode.
// Chmod changes the mode of the named file to mode.
Chmod(name string, mode os.FileMode) error
// Chown changes the uid and gid of the named file.
Chown(name string, uid, gid int) error
//Chtimes changes the access and modification times of the named file
Chtimes(name string, atime time.Time, mtime time.Time) error
}
+2 -2
View File
@@ -10,6 +10,6 @@ build_script:
go get -v github.com/spf13/afero/...
go build github.com/spf13/afero
go build -v github.com/spf13/afero/...
test_script:
- cmd: go test -race -v github.com/spf13/afero/...
- cmd: go test -count=1 -cover -race -v github.com/spf13/afero/...
+32 -1
View File
@@ -83,6 +83,13 @@ func (b *BasePathFs) Chmod(name string, mode os.FileMode) (err error) {
return b.source.Chmod(name, mode)
}
func (b *BasePathFs) Chown(name string, uid, gid int) (err error) {
if name, err = b.RealPath(name); err != nil {
return &os.PathError{Op: "chown", Path: name, Err: err}
}
return b.source.Chown(name, uid, gid)
}
func (b *BasePathFs) Name() string {
return "BasePathFs"
}
@@ -177,4 +184,28 @@ func (b *BasePathFs) LstatIfPossible(name string) (os.FileInfo, bool, error) {
return fi, false, err
}
// vim: ts=4 sw=4 noexpandtab nolist syn=go
func (b *BasePathFs) SymlinkIfPossible(oldname, newname string) error {
oldname, err := b.RealPath(oldname)
if err != nil {
return &os.LinkError{Op: "symlink", Old: oldname, New: newname, Err: err}
}
newname, err = b.RealPath(newname)
if err != nil {
return &os.LinkError{Op: "symlink", Old: oldname, New: newname, Err: err}
}
if linker, ok := b.source.(Linker); ok {
return linker.SymlinkIfPossible(oldname, newname)
}
return &os.LinkError{Op: "symlink", Old: oldname, New: newname, Err: ErrNoSymlink}
}
func (b *BasePathFs) ReadlinkIfPossible(name string) (string, error) {
name, err := b.RealPath(name)
if err != nil {
return "", &os.PathError{Op: "readlink", Path: name, Err: err}
}
if reader, ok := b.source.(LinkReader); ok {
return reader.ReadlinkIfPossible(name)
}
return "", &os.PathError{Op: "readlink", Path: name, Err: ErrNoReadlink}
}
+26 -1
View File
@@ -75,6 +75,10 @@ func (u *CacheOnReadFs) copyToLayer(name string) error {
return copyToLayer(u.base, u.layer, name)
}
func (u *CacheOnReadFs) copyFileToLayer(name string, flag int, perm os.FileMode) error {
return copyFileToLayer(u.base, u.layer, name, flag, perm)
}
func (u *CacheOnReadFs) Chtimes(name string, atime, mtime time.Time) error {
st, _, err := u.cacheStatus(name)
if err != nil {
@@ -117,6 +121,27 @@ func (u *CacheOnReadFs) Chmod(name string, mode os.FileMode) error {
return u.layer.Chmod(name, mode)
}
func (u *CacheOnReadFs) Chown(name string, uid, gid int) error {
st, _, err := u.cacheStatus(name)
if err != nil {
return err
}
switch st {
case cacheLocal:
case cacheHit:
err = u.base.Chown(name, uid, gid)
case cacheStale, cacheMiss:
if err := u.copyToLayer(name); err != nil {
return err
}
err = u.base.Chown(name, uid, gid)
}
if err != nil {
return err
}
return u.layer.Chown(name, uid, gid)
}
func (u *CacheOnReadFs) Stat(name string) (os.FileInfo, error) {
st, fi, err := u.cacheStatus(name)
if err != nil {
@@ -191,7 +216,7 @@ func (u *CacheOnReadFs) OpenFile(name string, flag int, perm os.FileMode) (File,
switch st {
case cacheLocal, cacheHit:
default:
if err := u.copyToLayer(name); err != nil {
if err := u.copyFileToLayer(name, flag, perm); err != nil {
return nil, err
}
}
+1 -1
View File
@@ -11,7 +11,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// +build darwin openbsd freebsd netbsd dragonfly
// +build aix darwin openbsd freebsd netbsd dragonfly
package afero
+1
View File
@@ -15,6 +15,7 @@
// +build !freebsd
// +build !dragonfly
// +build !netbsd
// +build !aix
package afero
+34 -1
View File
@@ -14,7 +14,7 @@ var _ Lstater = (*CopyOnWriteFs)(nil)
// a possibly writeable layer on top. Changes to the file system will only
// be made in the overlay: Changing an existing file in the base layer which
// is not present in the overlay will copy the file to the overlay ("changing"
// includes also calls to e.g. Chtimes() and Chmod()).
// includes also calls to e.g. Chtimes(), Chmod() and Chown()).
//
// Reading directories is currently only supported via Open(), not OpenFile().
type CopyOnWriteFs struct {
@@ -75,6 +75,19 @@ func (u *CopyOnWriteFs) Chmod(name string, mode os.FileMode) error {
return u.layer.Chmod(name, mode)
}
func (u *CopyOnWriteFs) Chown(name string, uid, gid int) error {
b, err := u.isBaseFile(name)
if err != nil {
return err
}
if b {
if err := u.copyToLayer(name); err != nil {
return err
}
}
return u.layer.Chown(name, uid, gid)
}
func (u *CopyOnWriteFs) Stat(name string) (os.FileInfo, error) {
fi, err := u.layer.Stat(name)
if err != nil {
@@ -117,6 +130,26 @@ func (u *CopyOnWriteFs) LstatIfPossible(name string) (os.FileInfo, bool, error)
return fi, false, err
}
func (u *CopyOnWriteFs) SymlinkIfPossible(oldname, newname string) error {
if slayer, ok := u.layer.(Linker); ok {
return slayer.SymlinkIfPossible(oldname, newname)
}
return &os.LinkError{Op: "symlink", Old: oldname, New: newname, Err: ErrNoSymlink}
}
func (u *CopyOnWriteFs) ReadlinkIfPossible(name string) (string, error) {
if rlayer, ok := u.layer.(LinkReader); ok {
return rlayer.ReadlinkIfPossible(name)
}
if rbase, ok := u.base.(LinkReader); ok {
return rbase.ReadlinkIfPossible(name)
}
return "", &os.PathError{Op: "readlink", Path: name, Err: ErrNoReadlink}
}
func (u *CopyOnWriteFs) isNotExist(err error) bool {
if e, ok := err.(*os.PathError); ok {
err = e.Err
-3
View File
@@ -1,3 +0,0 @@
module github.com/spf13/afero
require golang.org/x/text v0.3.0
-2
View File
@@ -1,2 +0,0 @@
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+4
View File
@@ -67,6 +67,10 @@ func (h HttpFs) Chmod(name string, mode os.FileMode) error {
return h.source.Chmod(name, mode)
}
func (h HttpFs) Chown(name string, uid, gid int) error {
return h.source.Chown(name, uid, gid)
}
func (h HttpFs) Chtimes(name string, atime time.Time, mtime time.Time) error {
return h.source.Chtimes(name, atime, mtime)
}
+288
View File
@@ -0,0 +1,288 @@
// +build go1.16
package afero
import (
"io"
"io/fs"
"os"
"path"
"time"
)
// IOFS adopts afero.Fs to stdlib io/fs.FS
type IOFS struct {
Fs
}
func NewIOFS(fs Fs) IOFS {
return IOFS{Fs: fs}
}
var (
_ fs.FS = IOFS{}
_ fs.GlobFS = IOFS{}
_ fs.ReadDirFS = IOFS{}
_ fs.ReadFileFS = IOFS{}
_ fs.StatFS = IOFS{}
_ fs.SubFS = IOFS{}
)
func (iofs IOFS) Open(name string) (fs.File, error) {
const op = "open"
// by convention for fs.FS implementations we should perform this check
if !fs.ValidPath(name) {
return nil, iofs.wrapError(op, name, fs.ErrInvalid)
}
file, err := iofs.Fs.Open(name)
if err != nil {
return nil, iofs.wrapError(op, name, err)
}
// file should implement fs.ReadDirFile
if _, ok := file.(fs.ReadDirFile); !ok {
file = readDirFile{file}
}
return file, nil
}
func (iofs IOFS) Glob(pattern string) ([]string, error) {
const op = "glob"
// afero.Glob does not perform this check but it's required for implementations
if _, err := path.Match(pattern, ""); err != nil {
return nil, iofs.wrapError(op, pattern, err)
}
items, err := Glob(iofs.Fs, pattern)
if err != nil {
return nil, iofs.wrapError(op, pattern, err)
}
return items, nil
}
func (iofs IOFS) ReadDir(name string) ([]fs.DirEntry, error) {
items, err := ReadDir(iofs.Fs, name)
if err != nil {
return nil, iofs.wrapError("readdir", name, err)
}
ret := make([]fs.DirEntry, len(items))
for i := range items {
ret[i] = dirEntry{items[i]}
}
return ret, nil
}
func (iofs IOFS) ReadFile(name string) ([]byte, error) {
const op = "readfile"
if !fs.ValidPath(name) {
return nil, iofs.wrapError(op, name, fs.ErrInvalid)
}
bytes, err := ReadFile(iofs.Fs, name)
if err != nil {
return nil, iofs.wrapError(op, name, err)
}
return bytes, nil
}
func (iofs IOFS) Sub(dir string) (fs.FS, error) { return IOFS{NewBasePathFs(iofs.Fs, dir)}, nil }
func (IOFS) wrapError(op, path string, err error) error {
if _, ok := err.(*fs.PathError); ok {
return err // don't need to wrap again
}
return &fs.PathError{
Op: op,
Path: path,
Err: err,
}
}
// dirEntry provides adapter from os.FileInfo to fs.DirEntry
type dirEntry struct {
fs.FileInfo
}
var _ fs.DirEntry = dirEntry{}
func (d dirEntry) Type() fs.FileMode { return d.FileInfo.Mode().Type() }
func (d dirEntry) Info() (fs.FileInfo, error) { return d.FileInfo, nil }
// readDirFile provides adapter from afero.File to fs.ReadDirFile needed for correct Open
type readDirFile struct {
File
}
var _ fs.ReadDirFile = readDirFile{}
func (r readDirFile) ReadDir(n int) ([]fs.DirEntry, error) {
items, err := r.File.Readdir(n)
if err != nil {
return nil, err
}
ret := make([]fs.DirEntry, len(items))
for i := range items {
ret[i] = dirEntry{items[i]}
}
return ret, nil
}
// FromIOFS adopts io/fs.FS to use it as afero.Fs
// Note that io/fs.FS is read-only so all mutating methods will return fs.PathError with fs.ErrPermission
// To store modifications you may use afero.CopyOnWriteFs
type FromIOFS struct {
fs.FS
}
var _ Fs = FromIOFS{}
func (f FromIOFS) Create(name string) (File, error) { return nil, notImplemented("create", name) }
func (f FromIOFS) Mkdir(name string, perm os.FileMode) error { return notImplemented("mkdir", name) }
func (f FromIOFS) MkdirAll(path string, perm os.FileMode) error {
return notImplemented("mkdirall", path)
}
func (f FromIOFS) Open(name string) (File, error) {
file, err := f.FS.Open(name)
if err != nil {
return nil, err
}
return fromIOFSFile{File: file, name: name}, nil
}
func (f FromIOFS) OpenFile(name string, flag int, perm os.FileMode) (File, error) {
return f.Open(name)
}
func (f FromIOFS) Remove(name string) error {
return notImplemented("remove", name)
}
func (f FromIOFS) RemoveAll(path string) error {
return notImplemented("removeall", path)
}
func (f FromIOFS) Rename(oldname, newname string) error {
return notImplemented("rename", oldname)
}
func (f FromIOFS) Stat(name string) (os.FileInfo, error) { return fs.Stat(f.FS, name) }
func (f FromIOFS) Name() string { return "fromiofs" }
func (f FromIOFS) Chmod(name string, mode os.FileMode) error {
return notImplemented("chmod", name)
}
func (f FromIOFS) Chown(name string, uid, gid int) error {
return notImplemented("chown", name)
}
func (f FromIOFS) Chtimes(name string, atime time.Time, mtime time.Time) error {
return notImplemented("chtimes", name)
}
type fromIOFSFile struct {
fs.File
name string
}
func (f fromIOFSFile) ReadAt(p []byte, off int64) (n int, err error) {
readerAt, ok := f.File.(io.ReaderAt)
if !ok {
return -1, notImplemented("readat", f.name)
}
return readerAt.ReadAt(p, off)
}
func (f fromIOFSFile) Seek(offset int64, whence int) (int64, error) {
seeker, ok := f.File.(io.Seeker)
if !ok {
return -1, notImplemented("seek", f.name)
}
return seeker.Seek(offset, whence)
}
func (f fromIOFSFile) Write(p []byte) (n int, err error) {
return -1, notImplemented("write", f.name)
}
func (f fromIOFSFile) WriteAt(p []byte, off int64) (n int, err error) {
return -1, notImplemented("writeat", f.name)
}
func (f fromIOFSFile) Name() string { return f.name }
func (f fromIOFSFile) Readdir(count int) ([]os.FileInfo, error) {
rdfile, ok := f.File.(fs.ReadDirFile)
if !ok {
return nil, notImplemented("readdir", f.name)
}
entries, err := rdfile.ReadDir(count)
if err != nil {
return nil, err
}
ret := make([]os.FileInfo, len(entries))
for i := range entries {
ret[i], err = entries[i].Info()
if err != nil {
return nil, err
}
}
return ret, nil
}
func (f fromIOFSFile) Readdirnames(n int) ([]string, error) {
rdfile, ok := f.File.(fs.ReadDirFile)
if !ok {
return nil, notImplemented("readdir", f.name)
}
entries, err := rdfile.ReadDir(n)
if err != nil {
return nil, err
}
ret := make([]string, len(entries))
for i := range entries {
ret[i] = entries[i].Name()
}
return ret, nil
}
func (f fromIOFSFile) Sync() error { return nil }
func (f fromIOFSFile) Truncate(size int64) error {
return notImplemented("truncate", f.name)
}
func (f fromIOFSFile) WriteString(s string) (ret int, err error) {
return -1, notImplemented("writestring", f.name)
}
func notImplemented(op, path string) error {
return &fs.PathError{Op: op, Path: path, Err: fs.ErrPermission}
}
+21 -11
View File
@@ -22,6 +22,7 @@ import (
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"time"
)
@@ -147,7 +148,7 @@ func reseed() uint32 {
return uint32(time.Now().UnixNano() + int64(os.Getpid()))
}
func nextSuffix() string {
func nextRandom() string {
randmu.Lock()
r := rand
if r == 0 {
@@ -159,27 +160,36 @@ func nextSuffix() string {
return strconv.Itoa(int(1e9 + r%1e9))[1:]
}
// TempFile creates a new temporary file in the directory dir
// with a name beginning with prefix, opens the file for reading
// and writing, and returns the resulting *File.
// TempFile creates a new temporary file in the directory dir,
// opens the file for reading and writing, and returns the resulting *os.File.
// The filename is generated by taking pattern and adding a random
// string to the end. If pattern includes a "*", the random string
// replaces the last "*".
// If dir is the empty string, TempFile uses the default directory
// for temporary files (see os.TempDir).
// Multiple programs calling TempFile simultaneously
// will not choose the same file. The caller can use f.Name()
// to find the pathname of the file. It is the caller's responsibility
// will not choose the same file. The caller can use f.Name()
// to find the pathname of the file. It is the caller's responsibility
// to remove the file when no longer needed.
func (a Afero) TempFile(dir, prefix string) (f File, err error) {
return TempFile(a.Fs, dir, prefix)
func (a Afero) TempFile(dir, pattern string) (f File, err error) {
return TempFile(a.Fs, dir, pattern)
}
func TempFile(fs Fs, dir, prefix string) (f File, err error) {
func TempFile(fs Fs, dir, pattern string) (f File, err error) {
if dir == "" {
dir = os.TempDir()
}
var prefix, suffix string
if pos := strings.LastIndex(pattern, "*"); pos != -1 {
prefix, suffix = pattern[:pos], pattern[pos+1:]
} else {
prefix = pattern
}
nconflict := 0
for i := 0; i < 10000; i++ {
name := filepath.Join(dir, prefix+nextSuffix())
name := filepath.Join(dir, prefix+nextRandom()+suffix)
f, err = fs.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0600)
if os.IsExist(err) {
if nconflict++; nconflict > 10 {
@@ -211,7 +221,7 @@ func TempDir(fs Fs, dir, prefix string) (name string, err error) {
nconflict := 0
for i := 0; i < 10000; i++ {
try := filepath.Join(dir, prefix+nextSuffix())
try := filepath.Join(dir, prefix+nextRandom())
err = fs.Mkdir(try, 0700)
if os.IsExist(err) {
if nconflict++; nconflict > 10 {
+1 -1
View File
@@ -106,5 +106,5 @@ func glob(fs Fs, dir, pattern string, matches []string) (m []string, e error) {
// recognized by Match.
func hasMeta(path string) bool {
// TODO(niemeyer): Should other magic characters be added here?
return strings.IndexAny(path, "*?[") >= 0
return strings.ContainsAny(path, "*?[")
}
+30 -9
View File
@@ -22,10 +22,9 @@ import (
"path/filepath"
"sync"
"sync/atomic"
"time"
)
import "time"
const FilePathSeparator = string(filepath.Separator)
type File struct {
@@ -57,6 +56,8 @@ type FileData struct {
dir bool
mode os.FileMode
modtime time.Time
uid int
gid int
}
func (d *FileData) Name() string {
@@ -95,6 +96,18 @@ func setModTime(f *FileData, mtime time.Time) {
f.modtime = mtime
}
func SetUID(f *FileData, uid int) {
f.Lock()
f.uid = uid
f.Unlock()
}
func SetGID(f *FileData, gid int) {
f.Lock()
f.gid = gid
f.Unlock()
}
func GetFileInfo(f *FileData) *FileInfo {
return &FileInfo{f}
}
@@ -193,8 +206,11 @@ func (f *File) Read(b []byte) (n int, err error) {
}
func (f *File) ReadAt(b []byte, off int64) (n int, err error) {
prev := atomic.LoadInt64(&f.at)
atomic.StoreInt64(&f.at, off)
return f.Read(b)
n, err = f.Read(b)
atomic.StoreInt64(&f.at, prev)
return
}
func (f *File) Truncate(size int64) error {
@@ -207,6 +223,8 @@ func (f *File) Truncate(size int64) error {
if size < 0 {
return ErrOutOfRange
}
f.fileData.Lock()
defer f.fileData.Unlock()
if size > int64(len(f.fileData.data)) {
diff := size - int64(len(f.fileData.data))
f.fileData.data = append(f.fileData.data, bytes.Repeat([]byte{00}, int(diff))...)
@@ -222,17 +240,20 @@ func (f *File) Seek(offset int64, whence int) (int64, error) {
return 0, ErrFileClosed
}
switch whence {
case 0:
case io.SeekStart:
atomic.StoreInt64(&f.at, offset)
case 1:
atomic.AddInt64(&f.at, int64(offset))
case 2:
case io.SeekCurrent:
atomic.AddInt64(&f.at, offset)
case io.SeekEnd:
atomic.StoreInt64(&f.at, int64(len(f.fileData.data))+offset)
}
return f.at, nil
}
func (f *File) Write(b []byte) (n int, err error) {
if f.closed == true {
return 0, ErrFileClosed
}
if f.readOnly {
return 0, &os.PathError{Op: "write", Path: f.fileData.name, Err: errors.New("file handle is read only")}
}
@@ -246,7 +267,7 @@ func (f *File) Write(b []byte) (n int, err error) {
tail = f.fileData.data[n+int(cur):]
}
if diff > 0 {
f.fileData.data = append(bytes.Repeat([]byte{00}, int(diff)), b...)
f.fileData.data = append(f.fileData.data, append(bytes.Repeat([]byte{00}, int(diff)), b...)...)
f.fileData.data = append(f.fileData.data, tail...)
} else {
f.fileData.data = append(f.fileData.data[:cur], b...)
@@ -254,7 +275,7 @@ func (f *File) Write(b []byte) (n int, err error) {
}
setModTime(f.fileData, time.Now())
atomic.StoreInt64(&f.at, int64(len(f.fileData.data)))
atomic.AddInt64(&f.at, int64(n))
return
}
+57 -18
View File
@@ -25,6 +25,8 @@ import (
"github.com/spf13/afero/mem"
)
const chmodBits = os.ModePerm | os.ModeSetuid | os.ModeSetgid | os.ModeSticky // Only a subset of bits are allowed to be changed. Documented under os.Chmod()
type MemMapFs struct {
mu sync.RWMutex
data map[string]*mem.FileData
@@ -40,7 +42,9 @@ func (m *MemMapFs) getData() map[string]*mem.FileData {
m.data = make(map[string]*mem.FileData)
// Root should always exist, right?
// TODO: what about windows?
m.data[FilePathSeparator] = mem.CreateDir(FilePathSeparator)
root := mem.CreateDir(FilePathSeparator)
mem.SetMode(root, os.ModeDir|0755)
m.data[FilePathSeparator] = root
})
return m.data
}
@@ -52,7 +56,7 @@ func (m *MemMapFs) Create(name string) (File, error) {
m.mu.Lock()
file := mem.CreateFile(name)
m.getData()[name] = file
m.registerWithParent(file)
m.registerWithParent(file, 0)
m.mu.Unlock()
return mem.NewFileHandle(file), nil
}
@@ -83,14 +87,14 @@ func (m *MemMapFs) findParent(f *mem.FileData) *mem.FileData {
return pfile
}
func (m *MemMapFs) registerWithParent(f *mem.FileData) {
func (m *MemMapFs) registerWithParent(f *mem.FileData, perm os.FileMode) {
if f == nil {
return
}
parent := m.findParent(f)
if parent == nil {
pdir := filepath.Dir(filepath.Clean(f.Name()))
err := m.lockfreeMkdir(pdir, 0777)
err := m.lockfreeMkdir(pdir, perm)
if err != nil {
//log.Println("Mkdir error:", err)
return
@@ -119,13 +123,15 @@ func (m *MemMapFs) lockfreeMkdir(name string, perm os.FileMode) error {
}
} else {
item := mem.CreateDir(name)
mem.SetMode(item, os.ModeDir|perm)
m.getData()[name] = item
m.registerWithParent(item)
m.registerWithParent(item, perm)
}
return nil
}
func (m *MemMapFs) Mkdir(name string, perm os.FileMode) error {
perm &= chmodBits
name = normalizePath(name)
m.mu.RLock()
@@ -137,13 +143,12 @@ func (m *MemMapFs) Mkdir(name string, perm os.FileMode) error {
m.mu.Lock()
item := mem.CreateDir(name)
mem.SetMode(item, os.ModeDir|perm)
m.getData()[name] = item
m.registerWithParent(item)
m.registerWithParent(item, perm)
m.mu.Unlock()
m.Chmod(name, perm|os.ModeDir)
return nil
return m.setFileMode(name, perm|os.ModeDir)
}
func (m *MemMapFs) MkdirAll(path string, perm os.FileMode) error {
@@ -210,8 +215,12 @@ func (m *MemMapFs) lockfreeOpen(name string) (*mem.FileData, error) {
}
func (m *MemMapFs) OpenFile(name string, flag int, perm os.FileMode) (File, error) {
perm &= chmodBits
chmod := false
file, err := m.openWrite(name)
if err == nil && (flag&os.O_EXCL > 0) {
return nil, &os.PathError{Op: "open", Path: name, Err: ErrFileExists}
}
if os.IsNotExist(err) && (flag&os.O_CREATE > 0) {
file, err = m.Create(name)
chmod = true
@@ -237,7 +246,7 @@ func (m *MemMapFs) OpenFile(name string, flag int, perm os.FileMode) (File, erro
}
}
if chmod {
m.Chmod(name, perm)
return file, m.setFileMode(name, perm)
}
return file, nil
}
@@ -269,7 +278,7 @@ func (m *MemMapFs) RemoveAll(path string) error {
m.mu.RLock()
defer m.mu.RUnlock()
for p, _ := range m.getData() {
for p := range m.getData() {
if strings.HasPrefix(p, path) {
m.mu.RUnlock()
m.mu.Lock()
@@ -299,7 +308,7 @@ func (m *MemMapFs) Rename(oldname, newname string) error {
delete(m.getData(), oldname)
mem.ChangeFileName(fileData, newname)
m.getData()[newname] = fileData
m.registerWithParent(fileData)
m.registerWithParent(fileData, 0)
m.mu.Unlock()
m.mu.RLock()
} else {
@@ -308,6 +317,11 @@ func (m *MemMapFs) Rename(oldname, newname string) error {
return nil
}
func (m *MemMapFs) LstatIfPossible(name string) (os.FileInfo, bool, error) {
fileInfo, err := m.Stat(name)
return fileInfo, false, err
}
func (m *MemMapFs) Stat(name string) (os.FileInfo, error) {
f, err := m.Open(name)
if err != nil {
@@ -318,6 +332,21 @@ func (m *MemMapFs) Stat(name string) (os.FileInfo, error) {
}
func (m *MemMapFs) Chmod(name string, mode os.FileMode) error {
mode &= chmodBits
m.mu.RLock()
f, ok := m.getData()[name]
m.mu.RUnlock()
if !ok {
return &os.PathError{Op: "chmod", Path: name, Err: ErrFileNotFound}
}
prevOtherBits := mem.GetFileInfo(f).Mode() & ^chmodBits
mode = prevOtherBits | mode
return m.setFileMode(name, mode)
}
func (m *MemMapFs) setFileMode(name string, mode os.FileMode) error {
name = normalizePath(name)
m.mu.RLock()
@@ -334,6 +363,22 @@ func (m *MemMapFs) Chmod(name string, mode os.FileMode) error {
return nil
}
func (m *MemMapFs) Chown(name string, uid, gid int) error {
name = normalizePath(name)
m.mu.RLock()
f, ok := m.getData()[name]
m.mu.RUnlock()
if !ok {
return &os.PathError{Op: "chown", Path: name, Err: ErrFileNotFound}
}
mem.SetUID(f, uid)
mem.SetGID(f, gid)
return nil
}
func (m *MemMapFs) Chtimes(name string, atime time.Time, mtime time.Time) error {
name = normalizePath(name)
@@ -357,9 +402,3 @@ func (m *MemMapFs) List() {
fmt.Println(x.Name(), y.Size())
}
}
// func debugMemMapList(fs Fs) {
// if x, ok := fs.(*MemMapFs); ok {
// x.List()
// }
// }
+12
View File
@@ -91,6 +91,10 @@ func (OsFs) Chmod(name string, mode os.FileMode) error {
return os.Chmod(name, mode)
}
func (OsFs) Chown(name string, uid, gid int) error {
return os.Chown(name, uid, gid)
}
func (OsFs) Chtimes(name string, atime time.Time, mtime time.Time) error {
return os.Chtimes(name, atime, mtime)
}
@@ -99,3 +103,11 @@ func (OsFs) LstatIfPossible(name string) (os.FileInfo, bool, error) {
fi, err := os.Lstat(name)
return fi, true, err
}
func (OsFs) SymlinkIfPossible(oldname, newname string) error {
return os.Symlink(oldname, newname)
}
func (OsFs) ReadlinkIfPossible(name string) (string, error) {
return os.Readlink(name)
}
+16
View File
@@ -28,6 +28,10 @@ func (r *ReadOnlyFs) Chmod(n string, m os.FileMode) error {
return syscall.EPERM
}
func (r *ReadOnlyFs) Chown(n string, uid, gid int) error {
return syscall.EPERM
}
func (r *ReadOnlyFs) Name() string {
return "ReadOnlyFilter"
}
@@ -44,6 +48,18 @@ func (r *ReadOnlyFs) LstatIfPossible(name string) (os.FileInfo, bool, error) {
return fi, false, err
}
func (r *ReadOnlyFs) SymlinkIfPossible(oldname, newname string) error {
return &os.LinkError{Op: "symlink", Old: oldname, New: newname, Err: ErrNoSymlink}
}
func (r *ReadOnlyFs) ReadlinkIfPossible(name string) (string, error) {
if srdr, ok := r.source.(LinkReader); ok {
return srdr.ReadlinkIfPossible(name)
}
return "", &os.PathError{Op: "readlink", Path: name, Err: ErrNoReadlink}
}
func (r *ReadOnlyFs) Rename(o, n string) error {
return syscall.EPERM
}
+10
View File
@@ -60,6 +60,13 @@ func (r *RegexpFs) Chmod(name string, mode os.FileMode) error {
return r.source.Chmod(name, mode)
}
func (r *RegexpFs) Chown(name string, uid, gid int) error {
if err := r.dirOrMatches(name); err != nil {
return err
}
return r.source.Chown(name, uid, gid)
}
func (r *RegexpFs) Name() string {
return "RegexpFs"
}
@@ -126,6 +133,9 @@ func (r *RegexpFs) Open(name string) (File, error) {
}
}
f, err := r.source.Open(name)
if err != nil {
return nil, err
}
return &RegexpFile{f: f, re: r.re}, nil
}
+55
View File
@@ -0,0 +1,55 @@
// Copyright © 2018 Steve Francia <spf@spf13.com>.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package afero
import (
"errors"
)
// Symlinker is an optional interface in Afero. It is only implemented by the
// filesystems saying so.
// It indicates support for 3 symlink related interfaces that implement the
// behaviors of the os methods:
// - Lstat
// - Symlink, and
// - Readlink
type Symlinker interface {
Lstater
Linker
LinkReader
}
// Linker is an optional interface in Afero. It is only implemented by the
// filesystems saying so.
// It will call Symlink if the filesystem itself is, or it delegates to, the os filesystem,
// or the filesystem otherwise supports Symlink's.
type Linker interface {
SymlinkIfPossible(oldname, newname string) error
}
// ErrNoSymlink is the error that will be wrapped in an os.LinkError if a file system
// does not support Symlink's either directly or through its delegated filesystem.
// As expressed by support for the Linker interface.
var ErrNoSymlink = errors.New("symlink not supported")
// LinkReader is an optional interface in Afero. It is only implemented by the
// filesystems saying so.
type LinkReader interface {
ReadlinkIfPossible(name string) (string, error)
}
// ErrNoReadlink is the error that will be wrapped in an os.Path if a file system
// does not support the readlink operation either directly or through its delegated filesystem.
// As expressed by support for the LinkReader interface.
var ErrNoReadlink = errors.New("readlink not supported")
+28 -17
View File
@@ -186,25 +186,22 @@ func (f *UnionFile) Readdir(c int) (ofi []os.FileInfo, err error) {
}
f.files = append(f.files, merged...)
}
files := f.files[f.off:]
if c <= 0 && len(f.files) == 0 {
return f.files, nil
if c <= 0 {
return files, nil
}
if f.off >= len(f.files) {
if len(files) == 0 {
return nil, io.EOF
}
if c <= 0 {
return f.files[f.off:], nil
}
if c > len(f.files) {
c = len(f.files)
if c > len(files) {
c = len(files)
}
defer func() { f.off += c }()
return f.files[f.off:c], nil
return files[:c], nil
}
func (f *UnionFile) Readdirnames(c int) ([]string, error) {
@@ -271,13 +268,7 @@ func (f *UnionFile) WriteString(s string) (n int, err error) {
return 0, BADFD
}
func copyToLayer(base Fs, layer Fs, name string) error {
bfh, err := base.Open(name)
if err != nil {
return err
}
defer bfh.Close()
func copyFile(base Fs, layer Fs, name string, bfh File) error {
// First make sure the directory exists
exists, err := Exists(layer, filepath.Dir(name))
if err != nil {
@@ -318,3 +309,23 @@ func copyToLayer(base Fs, layer Fs, name string) error {
}
return layer.Chtimes(name, bfi.ModTime(), bfi.ModTime())
}
func copyToLayer(base Fs, layer Fs, name string) error {
bfh, err := base.Open(name)
if err != nil {
return err
}
defer bfh.Close()
return copyFile(base, layer, name, bfh)
}
func copyFileToLayer(base Fs, layer Fs, name string, flag int, perm os.FileMode) error {
bfh, err := base.OpenFile(name, flag, perm)
if err != nil {
return err
}
defer bfh.Close()
return copyFile(base, layer, name, bfh)
}