Skip to main content

Fetch files with HTTPS

Files can be fetched with HTTP from files.unifaun.se.

Note

You should always use HTTPS.

There are two methods of authentication. The authentication method is specified with the “auth” parameter. If the “auth” parameter is omitted and no “Authorization” HTTP header is sent, the authentication is set to HTTP Digest.

HTTP Digest access authentication

The default authentication mode. The advantage is that the password is not transmitted unmodified to the web server which makes it safer than basic authentication.

Set the parameter “auth” to “DIGEST” or don't send the parameter.

Note

User name should be prefixed with “UO.”, for example, UO.002000111.

Please refer to https://tools.ietf.org/html/rfc2617.

HTTP Basic access authentication

User name and password are sent in the HTTP header.

Caution

This method is only safe if HTTPS is used.

Set the parameter “auth” to “BASIC”. The server will also try to automatically detect this authentication method using the HTTP header “Authorization”.

Please refer to https://www.w3schools.com/tags/ref_urlencode.asp.

Fetching files

  1. List the files in a directory

    Use an HTTP GET with the directory path as URL. The parameter view=SIMPLETEXT returns a result in plain text that is easy to interpret. URLs should use UTF-8 character set encoding and standard “URL encoding”. Each row in the result represents a file and is terminated by a line feed (0x0A). The encoding of the result is UTF-8.

    GET /files/?view=SIMPLETEXT HTTP/1.1

File1.txt
File2.txt
File3.txt

  2. Fetch files

    Use an HTTP GET with the file path as URL.

    GET /files/File1.txt?view=SIMPLETEXT HTTP/1.1

  3. Delete files

    Use an HTTP DELETE with the file path as URL.

    DELETE /files/File1.txt?view=SIMPLETEXT HTTP/1.1

    Another method for deleting the file is to use an HTTP POST with the file path as URL and a parameter “action” with the value “delete”:

    POST /files/File1.txt?view=SIMPLETEXT&action=delete HTTP/1.1

Using Wget

GNU Wget is a command line utility that is available both for Microsoft Windows and Unix-like operating systems.

Please refer to http://www.gnu.org/software/wget/ and http://gnuwin32.sourceforge.net/packages/wget.htm. The following script for Unix (bash shell) fetches all files that end with “.txt” from a directory and deletes the files from the source.

#!/bin/bash
HOST='user:password@files.unifaun.se'
DIRECTORY='out'
PARAMS='view=SIMPLETEXT'
FILES=$(wget -qO - "https://$HOST/$DIRECTORY/?$PARAMS" | grep -i ".txt$")
echo "Fetching files:" $FILES
for FILE in $FILES; do
  wget -qO - "https://$HOST/$DIRECTORY/$FILE" > $FILE && \
  wget –qO - "https://$HOST/$DIRECTORY/$FILE" \
    --post-data='action=delete' > /dev/null;
done

The command wget -nc -nd -r -l1 --no-parent -A "*.txt " \ will fetch all files in the “out” directory that end with “.txt” and haven't already been fetched.

https://user:pwd@files.unifaun.se/out/?view=SIMPLEHTML

Using curl

Curl is a command line utility that is available on several operating systems. Please refer to http://curl.haxx.se/.

The following script for Unix (Bash shell) fetches all files that end with “.txt” from a directory and deletes the files from the source.

#!/bin/bash
HOST='user:password@files.unifaun.se'
DIRECTORY='out'
PARAMS='view=SIMPLETEXT'
FILES=$(curl -sS "https://$HOST/$DIRECTORY/?$PARAMS" | grep -i ".txt$")
echo "Fetching files:" $FILES
for FILE in $FILES; do
  curl -OfsS "https://$HOST/$DIRECTORY/$FILE" && \
  curl -fsS "https://$HOST/$DIRECTORY/$FILE" \
    --form-string "action=delete" > /dev/null;
done

The following script for Windows (Powershell) fetches all files that end with “.txt” from a directory and deletes the files from the source.

$TARGETHOST="user:password@files.unifaun.se"
$DIRECTORY="out"
$PARAMS="view=SIMPLETEXT"
$FILES = @(curl -sS "https://$TARGETHOST/$DIRECTORY/?$PARAMS" `
  | select-string -pattern "\.txt$")
if ($FILES.length -gt 0) {
  write-host "Fetching files:" $FILES
  foreach ($FILE in $FILES) {
    curl -OfsS https://$TARGETHOST/$DIRECTORY/$FILE
    if ($?) {
      curl -fsS "https://$TARGETHOST/$DIRECTORY/$FILE" `
        --form-string "action=delete" > $null
    }
  }
}
In this section: