dot-files/nvim/scripts/test.readme.md

6. Improving the extract_title Function

a. Handling Complex YAML Frontmatter

Issue: YAML frontmatter can contain more complex structures, including quotes, multiple spaces, or special characters, which the current extract_title function might not handle correctly.

Solution: Enhance the title extraction to handle various YAML frontmatter formats more robustly.

Implementation:

extract_title() {
    local md_file="$1"
    # Extract title from YAML frontmatter, handling quotes and spaces
    local title
    title=$(awk '/^title:/ { 
        sub(/^title:\s*/, ""); 
        gsub(/['"'"']/,"", $0); 
        print 
        exit 
    }' "$md_file") || true
    if [[ -z "$title" ]]; then
        # Fallback to filename without extension
        title=$(basename "$md_file" .md.old)
    fi
    echo "$title"
}

Explanation:

• awk Usage:
  • Searches for a line starting with title:.
  • Removes the title: prefix and any leading whitespace.
  • Removes surrounding quotes if present.
  • Prints the title and exits to prevent processing the entire file.

---

10. Incorporating Configuration Flexibility

Issue:

Hardcoding paths and settings reduces the script's flexibility and adaptability to different environments.

Solution:

Allow configuration through external files or command-line arguments, making the script more versatile.

Implementation:

1. Using a Configuration File:

   • Create a Config File (config.sh):

     #!/usr/bin/env bash
     
     SOURCE_DIR="$HOME/vimwiki"
     TEMP_DIR="/tmp/vimwikihtml"
     CSS_FILE="$HOME/.local/share/nvim/style.css"
     CONCURRENT_JOBS=4
     LOG_FILE="$TEMP_DIR/conversion.log"
     ERROR_LOG_FILE="$TEMP_DIR/error.log"
     

   • Source the Config File in the Script:

     # At the beginning of test.sh
     if [[ -f "$HOME/vimwiki/config.sh" ]]; then
         source "$HOME/vimwiki/config.sh"
     else
         echo "Error: Configuration file 'config.sh' not found in '$HOME/vimwiki/'."
         exit 1
     fi
     

2. Using Command-Line Arguments:

   • Parse Arguments Using getopts:

     while getopts "s:t:c:j:l:e:" opt; do
         case $opt in
             s) SOURCE_DIR="$OPTARG" ;;
             t) TEMP_DIR="$OPTARG" ;;
             c) CSS_FILE="$OPTARG" ;;
             j) CONCURRENT_JOBS="$OPTARG" ;;
             l) LOG_FILE="$OPTARG" ;;
             e) ERROR_LOG_FILE="$OPTARG" ;;
             *) echo "Invalid option"; exit 1 ;;
         esac
     done
     

   • Usage Example:

     ./test.sh -s "/path/to/source" -t "/path/to/temp" -c "/path/to/style.css" -j 8
     

Benefits:

• Flexibility: Easily adapt to different directories, styles, and settings without modifying the script.
• Reusability: Share the script across different projects with varying configurations.

---

11. Adding a Dry-Run Mode for Safe Testing

Issue:

Making changes to files (like deletions) without a preview can be risky, especially during testing.

Solution:

Implement a dry-run mode that simulates actions without performing them, allowing you to verify behavior before actual execution.

Implementation:

1. Introduce a DRY_RUN Variable:

   DRY_RUN=false
   

2. Parse a Command-Line Argument for Dry-Run:

   while getopts "s:t:c:j:l:e:dr" opt; do
       case $opt in
           # ... [existing options]
           d) DRY_RUN=true ;;
           r) # Any other options
              ;;
           *) echo "Invalid option"; exit 1 ;;
       esac
   done
   

3. Modify File Operations to Respect Dry-Run:

   handle_deletions() {
       echo "Handling deletions of Markdown files..."
   
       find "$TEMP_DIR" -type f -name '*.md.old' -print0 | while IFS= read -r -d '' md_old_file; do
           source_md="$SOURCE_DIR/${md_old_file#$TEMP_DIR/}"
           source_md="${source_md%.md.old}.md"
   
           if [[ ! -f "$source_md" ]]; then
               html_file="${md_old_file%.md.old}.html"
   
               if [[ "$DRY_RUN" = true ]]; then
                   echo "Dry-Run: Would delete '$html_file' as the source Markdown file no longer exists."
               else
                   if [[ -f "$html_file" ]]; then
                       rm "$html_file"
                       echo "Deleted '$html_file' as the source Markdown file no longer exists."
                       echo "$(date +"%Y-%m-%d %H:%M:%S") - Deleted '$html_file' due to source removal." >> "$LOG_FILE"
                   fi
   
                   rm "$md_old_file"
                   echo "Removed obsolete '$md_old_file'."
               fi
           fi
       done
   
       echo "Deletion handling completed."
   }
   

4. Inform Users When in Dry-Run Mode:

   if [[ "$DRY_RUN" = true ]]; then
       echo "Running in Dry-Run mode. No changes will be made."
   fi
   

Benefits:

• Safety: Allows you to verify what actions the script would take without making actual changes.
• Testing: Facilitates safe testing and debugging of the script's logic.

---

Explanation:

• trap cleanup SIGINT SIGTERM: Catches interrupt (Ctrl+C) and termination signals, invoking the cleanup function.
• cleanup Function:
  • Echoes a message indicating interruption.
  • Terminates all background jobs to prevent orphan processes.
  • Exits the script with a non-zero status.

---

13. Providing User Feedback and Progress Indicators

Issue:

With large numbers of files, users might find it hard to gauge progress or know if the script is still running.

Solution:

Implement progress indicators or verbose outputs to inform users about the script's status.

Implementation:

1. Using Progress Bars with pv:

   Install pv:

   sudo apt-get install pv
   

2. Integrate pv into File Processing:

   convert_all_md_to_html() {
       echo "Starting conversion of updated Markdown files to HTML..."
   
       find "$TEMP_DIR" -type f -name '*.md.old' -print0 | pv -0 -l -s "$(find "$TEMP_DIR" -type f -name '*.md.old' | wc -l)" | parallel -0 -j "$CONCURRENT_JOBS" convert_md_to_html {}
   
       echo "Markdown to HTML conversion completed."
   }
   

3. Simple Progress Indicators:

   Alternatively, use echo statements to indicate progress.

   convert_md_to_html() {
       local md_old_file="$1"
       echo "Processing file: $md_old_file"
       # ... [rest of the function]
   }
   

Benefits:

• User Awareness: Users are informed about the script's progress, enhancing usability.
• Debugging: Progress indicators can help identify if the script is stuck or processing unusually.

---

14. Adding a Help Message and Usage Instructions

Issue:

Users might not be aware of available options or how to use the script effectively.

Solution:

Implement a help message that outlines usage instructions and available options.

Implementation:

print_help() {
    echo "Usage: $0 [options]"
    echo ""
    echo "Options:"
    echo "  -s DIR   Source Vimwiki directory (default: \$HOME/vimwiki)"
    echo "  -t DIR   Temporary HTML output directory (default: /tmp/vimwikihtml)"
    echo "  -c FILE  Path to the CSS file for styling (default: \$HOME/.local/share/nvim/style.css)"
    echo "  -j NUM   Number of concurrent pandoc processes (default: 4)"
    echo "  -l FILE  Log file to track conversions (default: /tmp/vimwikihtml/conversion.log)"
    echo "  -e FILE  Log file to track errors (default: /tmp/vimwikihtml/error.log)"
    echo "  -d       Enable Dry-Run mode"
    echo "  -h       Display this help message"
    echo ""
    echo "Example:"
    echo "  $0 -s ~/myvimwiki -t ~/htmloutput -c ~/styles/style.css -j 8"
}

Add argument parsing to handle the -h option:

# Parse command-line arguments
while getopts "s:t:c:j:l:e:dh" opt; do
    case $opt in
        s) SOURCE_DIR="$OPTARG" ;;
        t) TEMP_DIR="$OPTARG" ;;
        c) CSS_FILE="$OPTARG" ;;
        j) CONCURRENT_JOBS="$OPTARG" ;;
        l) LOG_FILE="$OPTARG" ;;
        e) ERROR_LOG_FILE="$OPTARG" ;;
        d) DRY_RUN=true ;;
        h)
            print_help
            exit 0
            ;;
        *)
            echo "Invalid option."
            print_help
            exit 1
            ;;
    esac
done

Benefits:

• Usability: Makes the script more user-friendly and accessible.
• Documentation: Provides immediate reference without external documentation.

---

17. Adding Verbose and Quiet Modes

Issue:

Sometimes users might prefer minimal output, especially when running the script in automated environments, while other times they might want detailed logs.

Solution:

Implement verbose (-v) and quiet (-q) modes to control the script's output.

Implementation:

1. Introduce Variables:

   VERBOSE=false
   QUIET=false
   

2. Update Argument Parsing:

   while getopts "s:t:c:j:l:e:dhvq" opt; do
       case $opt in
           # ... [existing options]
           v) VERBOSE=true ;;
           q) QUIET=true ;;
           # ... [other options]
       esac
   done
   

3. Modify Echo Statements:

   log() {
       if [[ "$VERBOSE" = true && "$QUIET" = false ]]; then
           echo "$@"
       fi
   }
   
   # Replace echo with log where appropriate
   log "Starting conversion of '$md_old_file'..."
   

4. Suppress Non-Essential Output in Quiet Mode:

   if [[ "$QUIET" = true ]]; then
       exec >/dev/null 2>&1
   fi
   

Benefits:

• Flexibility: Users can choose the level of output based on their needs.
• Usability: Enhances the script's adaptability in different contexts.

---

18. Documentation and In-Line Comments

Issue:

As scripts grow in complexity, maintaining clear documentation and comments becomes essential for future maintenance and collaboration.

Solution:

Add comprehensive in-line comments explaining the purpose and functionality of code sections and functions.

Implementation:

# Function to convert a single Markdown file to HTML
# Arguments:
#   $1 - Path to the .md.old file
convert_md_to_html() {
    # ... [function code]
}

Recommendation: Regularly update comments to reflect any changes in the script's logic or functionality.

---

19. Testing and Validation

Issue:

Ensuring that the script behaves as expected in various scenarios is crucial for reliability.

Solution:

Implement automated tests or conduct thorough manual testing covering different cases like:

• Adding new files.
• Modifying existing files.
• Deleting files.
• Handling files with special characters.
• Running in dry-run mode.
• Handling permission issues.

Implementation:

1. Automated Testing:

   • Use Bash Unit Testing Frameworks: Tools like Bats can help write automated tests for your script.

   • Example Test Case:

     @test "Convert new Markdown file to HTML" {
         # Setup: Create a new Markdown file in SOURCE_DIR
         echo "# Test Page" > "$SOURCE_DIR/test.md"
     
         # Run the script
         ~/vimwiki/test.sh
     
         # Assert: Check if HTML file exists
         [ -f "$TEMP_DIR/test.html" ]
     
         # Cleanup
         rm "$SOURCE_DIR/test.md" "$TEMP_DIR/test.html" "$TEMP_DIR/test.md.old"
     }
     

2. Manual Testing:

   • Create Test Files: Set up various Markdown files with different content and filenames.
   • Run the Script: Execute the script and verify the HTML output.
   • Check Logs: Ensure that logs accurately reflect the actions taken.
   • Edge Cases: Test with empty files, very large files, and files with complex YAML frontmatter.

Benefits:

• Reliability: Ensures that the script performs correctly under various conditions.
• Confidence: Builds trust in the script's functionality before deploying it in critical environments.

---

20. Additional Feature Suggestions

a. Watch Mode for Real-Time Updates

Issue: Manually running the script after every change can be tedious.

Solution: Implement a watch mode that monitors the source directory for changes and triggers the script automatically.

Implementation:

1. Use inotifywait:

   Install inotify-tools:

   sudo apt-get install inotify-tools
   

2. Add a Watch Mode Option:

   WATCH_MODE=false
   
   while getopts "s:t:c:j:l:e:dhvqw" opt; do
       case $opt in
           # ... [existing options]
           w) WATCH_MODE=true ;;
           # ... [other options]
       esac
   done
   

3. Implement Watch Mode in the Script:

   watch_mode() {
       echo "Entering Watch Mode. Monitoring '$SOURCE_DIR' for changes..."
       while inotifywait -r -e modify,create,delete,move "$SOURCE_DIR"; do
           echo "Change detected. Updating Vimwiki HTML..."
           update_if_needed
           open_browser
       done
   }
   
   main() {
       check_dependencies
   
       if [[ "$WATCH_MODE" = true ]]; then
           # Initial synchronization
           if [[ ! -d "$TEMP_DIR" ]]; then
               mkdir -p "$TEMP_DIR"
               synchronize_markdown
               rename_md_files
               handle_deletions
               generate_index
           else
               update_if_needed
           fi
   
           # Enter watch mode
           watch_mode
       else
           # Regular execution
           if [[ ! -d "$TEMP_DIR" ]]; then
               echo "Temporary directory '$TEMP_DIR' does not exist. Creating and performing full synchronization."
               mkdir -p "$TEMP_DIR"
               synchronize_markdown
               rename_md_files
               handle_deletions
               generate_index
           else
               echo "Temporary directory '$TEMP_DIR' already exists. Checking for updates."
               update_if_needed
           fi
   
           open_browser
   
           echo "All tasks completed successfully."
       fi
   }
   

Benefits:

• Convenience: Automatically keeps the HTML output up-to-date in real-time.
• Efficiency: Eliminates the need for manual script execution after each change.

Note: Watch mode can consume more resources. Use it judiciously based on your system's capabilities.

---

22. Implementing a Configuration Validation Step

Issue:

Incorrect configurations (like wrong directory paths) can lead to script failures or unintended behavior.

Solution:

Validate configuration parameters at the beginning of the script to ensure they are correct.

Implementation:

validate_config() {
    # Validate SOURCE_DIR
    if [[ ! -d "$SOURCE_DIR" ]]; then
        handle_error "Source directory '$SOURCE_DIR' does not exist."
        exit "$EXIT_FAILURE"
    fi
    
    # Validate CSS_FILE
    if [[ ! -f "$CSS_FILE" ]]; then
        echo "Warning: CSS file '$CSS_FILE' does not exist. HTML files will be generated without styling."
    fi
    
    # Validate TEMP_DIR (already handled in main)
}

main() {
    check_dependencies
    validate_config
    
    # ... [rest of the main function]
}

Benefits:

• Prevention: Catches configuration issues early, preventing the script from running with invalid settings.
• User Guidance: Provides clear error messages to help users correct configurations.

---

23. Utilizing set -x for Debugging

Issue:

Debugging complex scripts can be challenging without visibility into their execution flow.

Solution:

Use Bash's debugging options to trace the script's execution when needed.

Implementation:

1. Add a Verbose Flag (-x):

   set -euo pipefail
   [[ "$VERBOSE" = true ]] && set -x
   

2. Enable Verbose Mode via Command-Line Argument:

   while getopts "s:t:c:j:l:e:dhvqx" opt; do
       case $opt in
           # ... [existing options]
           x) VERBOSE=true ;;
           # ... [other options]
       esac
   done
   

Benefits:

• Transparency: Provides detailed execution logs for debugging purposes.
• Control: Users can enable or disable debugging as needed without modifying the script.

Caution: Ensure that verbose mode does not expose sensitive information, especially when handling logs.

---

24. Preventing Infinite Loops or Excessive Resource Consumption

Issue:

If the script triggers file changes (like converting .md.old to .html), it might inadvertently cause itself to detect changes, leading to infinite loops or excessive resource usage.

Solution:

Ensure that the script only processes intended file types and excludes its own output from triggering further actions.

Implementation:

1. Exclude .html and .md.old Files from Synchronization:

   • Already handled via rsync excludes.

2. Ensure handle_deletions and Other Functions Do Not Modify Source Directory:

   • All file operations target the temporary directory, ensuring the source remains untouched.

3. Double-Check rsync Include/Exclude Patterns:

   rsync -av --delete \
       --exclude='*.html' \
       --exclude='*.sh' \
       --exclude='.git/' \
       --exclude='.gitignore' \
       --exclude='*.bak' \
       --include='*/' \
       --include='*.md' \
       --include='*.pdf' \
       --include='*.png' \
       --include='*.jpg' \
       --include='*.jpeg' \
       --include='*.gif' \
       --exclude='*' \
       "$SOURCE_DIR/" "$TEMP_DIR/"
   

Recommendation: Regularly review and test the script to ensure it doesn't inadvertently process or modify unintended files.

---